Agent Observability: The Telemetry Gap Killing Your Agentic Stack

The hardest bugs in multi-agent systems aren't the ones that throw exceptions. They're the ones where an agent confidently returned a 200, the orchestrator marked the task complete, and the downstream record was silently wrong for three days before anyone noticed. Standard APM can't catch this. Most teams don't realize it until they're already in damage control.

This isn't a monitoring gap. It's an architectural assumption: that agent systems behave like request/response services and can be observed the same way. They can't. Let me show you exactly what to wire up — and walk through a concrete failure scenario to show how this telemetry actually catches the problem.

Why Standard APM Misses the Agent Problem Entirely

Conventional observability — Datadog, New Relic, CloudWatch — was built for request/response services. You have an input, a handler, a latency number, a status code. Done.

Agent systems don't work like that. A single user request might trigger:

• Three sub-agents running concurrently
• Tool calls firing in non-deterministic order
• LLM completions with variable context window sizes
• Mid-chain state mutations that affect subsequent agent behavior
• Retry loops when tool calls fail or return unexpected schemas

None of that structure is visible in a standard trace. You'll see a long HTTP request, maybe a high p95, and nothing else. When something goes wrong — wrong booking, wrong data routed to the wrong endpoint, hallucinated intermediate output used as ground truth — you're debugging blind.

Standard APM gives you a latency histogram. What you need is a reasoning trace: every decision the agent made, what context it had at that point, which tool it called, what the tool returned, and what the agent inferred from that return before passing state to the next node.

The Five Telemetry Dimensions Agents Actually Need

Here's the framework I use to evaluate whether an agentic stack has real observability or just the illusion of it.

1. Span-level agent traces with parent-child relationships Every agent invocation should emit an OpenTelemetry-compatible span. Sub-agents are child spans of the orchestrator span. Tool calls are child spans of the agent span. This gives you a DAG you can traverse post-hoc. Without this, you cannot reconstruct what happened in a multi-agent execution — you're left with a flat list of events that tells you nothing about causality.

2. Prompt and completion logging (with token accounting) Log the full prompt sent to the model and the full completion returned — not just the parsed output. Teams that skip this have no way to diagnose prompt injection, context truncation, or instruction drift. Tag every log entry with input token count, output token count, model ID, and temperature. This data is also your cost attribution layer: without it, you cannot answer which agent is responsible for your token bill.

3. Tool call input/output capture Every tool call should log: the function name, the exact arguments passed, the raw response, and the latency. This sounds obvious but many frameworks log only success/failure. When an agent passes malformed arguments to a tool, you need to see exactly what it sent — not just that a 400 came back. Frameworks like LangSmith, Langfuse, and Arize Phoenix make this easier, but the discipline of capturing it is independent of the tool.

4. State snapshots at handoff boundaries In a multi-agent system, state passes between agents. That boundary is where most silent failures happen. Serialize the inter-agent context at every handoff and store it with the trace ID. This alone will catch a category of bugs that currently live undetected in production — truncated fields, type coercions, schema mismatches that only appear at the boundary between one agent's output format and another's expected input.

5. Outcome correlation and ground-truth labeling This is the layer almost no team has. Telemetry is only useful if you can tie agent behavior to downstream outcomes. Did the booking complete correctly? Did the support ticket get resolved? Did the data transformation produce a valid record? You need a feedback loop that labels agent traces as successful or failed based on verifiable downstream state — not just "the agent returned a 200."

Without outcome correlation, you have logging. With it, you have observability.

A Minimal Telemetry Schema

Here's a schema worth wiring into any multi-agent system from day one. This is a starting point, not an exhaustive spec — but if you don't have these fields, you're flying blind.

json
{
  "trace_id": "uuid",
  "span_id": "uuid",
  "parent_span_id": "uuid | null",
  "agent_name": "string",
  "agent_version": "string",
  "model_id": "string",
  "timestamp_start": "ISO8601",
  "timestamp_end": "ISO8601",
  "input_tokens": "integer",
  "output_tokens": "integer",
  "prompt_hash": "sha256",
  "tool_calls": [
    {
      "tool_name": "string",
      "args": "object",
      "response": "object",
      "latency_ms": "integer",
      "status": "success | error | timeout"
    }
  ],
  "handoff_state_snapshot": "object | null",
  "completion_raw": "string",
  "outcome_label": "success | failure | unknown",
  "session_id": "string"
}

The prompt_hash field deserves a note: storing full prompts at scale is expensive. Hash them instead, store the full prompt only when the hash is new, and retrieve by hash when debugging. You get full fidelity at a fraction of the storage cost.

Diagnosing a Real Production Failure With This Schema

Abstract schemas don't teach you much. Here's how this telemetry actually surfaces a failure.

Imagine a travel booking pipeline with three agents: an intent classifier, a search agent, and a booking agent. A user requests a flight from Dubai to London, return on a specific date. The booking agent confirms success. The ticket is issued for the wrong return date.

Without this telemetry, you're interviewing the user and guessing. With it, here's the diagnostic path:

Step 1 — Pull the trace by session ID. You immediately see three child spans under the orchestrator span: intent_classifier, search_agent, booking_agent. Latencies look normal. No errors anywhere. The outcome_label field, populated by the downstream ticketing system's webhook, reads failure.

Step 2 — Inspect the handoff state snapshot at the search → booking boundary. The snapshot shows the return date field serialized as "2025-07-20T00:00:00Z". Correct. But the booking agent's tool_calls log shows it called the reservations API with return_date: "2025-20-07" — day and month transposed. The agent's tool wrapper was formatting dates with a locale-sensitive formatter that flips format under a specific timezone offset. The raw completion_raw log shows the model output the date correctly; the bug is in the tool wrapper, not the model.

Step 3 — Check prompt hash. The hash is identical to 47 prior runs. That means this isn't a prompt regression — it's an environment issue in the tool layer, probably a recent deployment that changed the server locale.

Total diagnosis time with this schema in place: under 10 minutes. Without it: days of log-grepping, customer interviews, and guesswork — if you find it at all before it happens again.

This is the gap. The schema fields aren't clever. The discipline of capturing them before you need them is.

The Cost Blindness Problem Is a Telemetry Problem

Without per-agent token accounting, you cannot govern your AI cost — and in agentic systems, cost is non-deterministic by design. An orchestrator that spawns sub-agents based on task complexity will have dramatically different token consumption depending on the input. That's expected behavior. What's not acceptable is not knowing which agent, which task type, and which tool pattern is driving your bill.

Imagine a team that discovers their support agent pipeline costs significantly more on certain ticket categories than others. Without trace-level token attribution, they can't find the root cause. They'd have to instrument from scratch while already in production, under pressure. With the schema above already running, it's a single aggregation query: group input_tokens + output_tokens by agent_name and filter by ticket category tag. The expensive agent surfaces immediately.

This is why token accounting belongs in the telemetry schema from day one, not as a billing afterthought.

The Replay Requirement

Here's the capability that separates teams who can actually improve their agent system from teams who can only react to it: deterministic replay.

Given a trace ID, can you replay that exact agent execution — same prompt, same tool responses, same state — and observe different behavior if you change the model or the prompt? If not, you cannot run controlled experiments on your production system. You're guessing.

Building replay is a three-part problem:

1. Input capture: store all external inputs (user message, tool responses, API returns) at the trace level.
2. Tool mocking: your tool layer needs to support a mock mode that replays stored responses instead of calling live APIs.
3. State seeding: inject the captured session state rather than initializing fresh.

This isn't exotic infrastructure. It's the same pattern as cassette-based HTTP mocking (VCR, Betamax) applied to agent execution. The engineering lift is days if you architect for it from the start. Retrofitting it to a system that wasn't built with capture in mind is where it gets expensive — and that cost is paid in engineering hours and in every production incident that happens in the meantime.

The Alerting Layer Most Teams Build Last (and Shouldn't)

Observability without alerting is just an expensive log archive. The signals worth alerting on in agentic systems are different from standard services:

Signal	Why It Matters	Alert Threshold Pattern
Tool call error rate per agent	Indicates schema drift or API change	> 5% over 5-min window
Average chain depth spike	Agents looping unexpectedly	2× baseline depth
Token/task ratio anomaly	Model behavior change or prompt drift	> 2 std dev from 7-day mean
Handoff state size growth	Context bloat degrading downstream agents	> 80% of model context window
Outcome label failure rate	Actual task failure, not just errors	> threshold per use case

The chain depth alert is the one I'd prioritize first. An agent that's supposed to execute in 3–5 steps and is now taking 15 is either stuck in a retry loop, caught in a reasoning spiral, or hitting a tool that keeps returning unexpected output. That's a runaway cost event and potentially a correctness event. You want to catch it in seconds, not after the fact — and you can't catch it at all without span-level tracing.

The Contrarian Truth About Observability and Agent Quality

Teams treat observability as infrastructure work — something you add after the system proves itself in production. This is exactly backwards for agentic systems. In a traditional API, a bug produces an error. In an agent system, a bug produces a confident, well-formatted wrong answer. The system appears healthy. The outcome is not.

Observability isn't what you add when things break. It's what lets you tell the difference between "working" and "appearing to work." In agentic AI, that distinction is the whole game.

What to Actually Do

• This week: instrument your agent spans with OpenTelemetry parent-child relationships. If you can't reconstruct execution order from your traces today, that's your first fix.
• This week: add the minimal schema above to every agent invocation. The prompt_hash, tool_calls, and handoff_state_snapshot fields alone will surface most silent failures.
• Next sprint: wire a downstream outcome webhook that writes outcome_label back to your trace store. This is the step that converts your logging into actual observability.
• Next sprint: implement tool mock mode and test replay on your three most complex traces from last week. If replay diverges unexpectedly, you've already found a non-determinism bug worth fixing.
• Ongoing: build chain depth and outcome failure rate alerts before you scale agent traffic, not after. The cost of a runaway agent loop at low traffic is annoying. At production scale it's a budget event.

You can ship a well-instrumented agentic system in days. The thing that takes weeks is explaining to stakeholders why a confident agent quietly failed for three days before anyone noticed.