A governed agent runtime built around inspectability — OpenTelemetry-shaped traces, bounded retries, a policy-gate layer that denies unsafe tool calls, and a documented failure-mode catalog — designed to run on a laptop and audit, not a tracing dashboard with agents bolted on.

The runtime ships v1 recorded runs on main. The canonical run runs/2026-05-06_240d1c56_0/ walks search → fetch → read → summarize → final_answer end-to-end via make canonical, against the synthetic 25-document fixture corpus and a deterministic stub LLM (src/runtime/stub_llm/) that keeps the canonical run hosted-LLM-free.

Seven policy-gate runs under runs/policy_gates/ and five failure-mode runs under runs/failure_modes/ complete the v1 recorded-run set. Each committed run carries the four required artifacts: trace.json, state.jsonl, run_report.md, and manifest.json.

The canonical run (2026-05-06_240d1c56_0) executes 5 agent steps with 4 tool calls and 4 retry attempts (0 exhausted; all retries succeeded), emits 22 trace spans (agent_step ×5, llm_call ×5, tool_call ×4, policy_check ×4, retry_attempt ×4), and terminates with final_answer against the 25-document synthetic fixture corpus. Of the five named policy-gate scenarios, four (PG1, PG2, PG4, PG5) fire deny spans with the documented rule_id set in the trace; PG3 (loop budget; both iteration and token-budget variants) terminates via agent.terminal_reason = loop_budget rather than a deny span — a known gap, tracked at runs/policy_gates/pg3_loop_budget/run_report.md. All five canonical failure modes fire agent.failure_mode on the offending step and reach their documented terminal state. Numbers and structural facts come from the captured run_report.md and manifest.json files; nothing in this prose claims more than those files support.

The runtime, policy, trace, tool, fixture, and evidence slices verify the runtime / tool boundary with:

• pytest tests/
• make smoke — verify file structure
• make smoke-runtime — runtime-skeleton tests
• make policy-gates — exercise the seven policy-gate scenarios; SCENARIO=<id> selects one
• make policy-gates-check — re-emit and diff each policy-gate run against its committed artifacts
• make failure-modes — exercise the five canonical modes; SCENARIO=<id> selects one
• make failure-modes-check — re-emit and diff each failure-mode run against its committed artifacts
• make trace-smoke — drive the in-tree trace fixture through the OTLP-JSON exporter and validate against the subset schema
• make tool-smoke — drive the five v1 tools through a real Agent.run with strict-mode arg_schema enforcement
• make fixture-build — (re)build the deterministic fixture corpus from the documented seed
• python3 -m scripts.build_fixture_corpus --check — verify on-disk fixture matches the manifest
• make canonical — drive the canonical task fixture through a real Agent.run against the fixture corpus and the v1 tool, policy, and trace surfaces
• make canonical-check — re-emit and diff the canonical run against the committed artifacts
• make evidence-build and make evidence-check — aggregate emit / diff across canonical + policy-gate + failure-mode runs

Scope: this is a bounded reference implementation for governed tool use, schema enforcement, policy gates, failure classification, and traceable agent execution — not a production agent platform.

git clone https://github.com/Ariel-J-Lee/agent-runtime-observability.git
cd agent-runtime-observability
pip install -r requirements.txt -r requirements-dev.txt
make smoke               # verify file structure
make smoke-runtime       # runtime-skeleton tests
make policy-gates        # exercise all seven policy-gate scenarios
make failure-modes       # exercise all five failure-mode triggers
make trace-smoke         # trace exporter + subset-schema validation
make tool-smoke          # five tools through a real Agent.run
make fixture-build       # (re)build the deterministic fixture corpus from the documented seed
python3 -m scripts.build_fixture_corpus --check   # verify on-disk fixture matches the manifest
make canonical           # canonical task through Agent.run on the fixture corpus
make canonical-check     # re-emit and diff the canonical run vs committed artifacts
make policy-gates-check  # re-emit and diff all seven policy-gate runs
make failure-modes-check # re-emit and diff all five failure-mode runs

The canonical run is deterministic given the pinned seed, the deterministic stub LLM, and the policy-spec hash. Two reviewers running make canonical-check against the same code state produce byte-identical trace.json, state.jsonl, and run_report.md. The manifest.json is byte-identical except on three documented per-run-volatile keys (timestamp, wall_clock_seconds, code.git_sha) that are excluded from the reproducibility diff.

This is a reference implementation, not a benchmark.

• Deterministic stub LLM canonical default. The committed runs use a deterministic stub LLM at src/runtime/stub_llm/canned.py, keyed by the canonical fixture. The Agent constructor accepts a Callable[[LLMInput], LLMOutput] seam, so a caller can plug in a live local-LLM or hosted-API adapter that satisfies that shape. No live local-LLM or hosted-API adapter ships in this release; the recorded runs use the stub default exclusively.
• Synthetic public-safe fixture corpus. Twenty-five hand-authored CC0-1.0 documents under data/corpus/v1/. The runtime is exercised against these only; results do not generalize beyond this corpus.
• Single-agent only at v1. No multi-agent coordination. Multi-agent claims do not appear anywhere in this README, docs/, or the repo description.
• Filesystem-path-based sandbox. Sandbox isolation is realpath-based against an allowlisted per-run directory. Not container-isolated and not capability-restricted at v1.
• PG3 known gap. The loop-budget policy (PG3 and PG3-tokens) terminates via agent.terminal_reason = loop_budget rather than emitting a policy_check deny span. The gap is locked in tests/evidence/test_pg3_runs_terminate_with_loop_budget_reason. Surfacing the deny span on PG3 is a follow-on runtime fix.
• Scope: reference, not benchmark. The captured runs demonstrate that the runtime components, the policy-gate denials, and the failure-mode triggers fire end-to-end with reproducible traces. They are not a performance, latency, throughput, or accuracy benchmark.
• Reproducibility envelope is bounded. Two reviewers running the same make canonical-check invocation produce byte-identical trace.json and state.jsonl given the pinned policy-spec hash, stub-LLM script hash, code SHA, and seed. Reproducibility across substantively different code or policy revisions is not claimed.
• No coding-agent claim, no MCP-server claim. Both deferred; neither is in v1 scope.
• No production deployment claim, no customer-deployment claim, no autonomous-agent operation, no large-scale inference platform claim, no RLHF / DPO / LoRA training. This is a local reference implementation.

Apache-2.0.

• production-rag-eval-harness — retrieval-quality evaluation harness.
• aws-bedrock-iac-reference — AWS / Bedrock infrastructure-as-code reference.

Cross-references are descriptive only; this repository does not import or deploy them.