Agent Observability: If You Can't Replay It, You Can't Debug It

My agent broke something in production. Not catastrophically — it generated a wrong API response that a downstream service silently accepted. By the time someone noticed, three hours had passed.

I couldn't tell you why it happened. I knew the model had been called. I knew a tool had run. But I didn't know what the model saw, what it considered, or why it chose the action it chose. My logs said "tool: edit_file, status: success." That's like a flight recorder that only records "plane: flew."

Every action becomes a trace event

The pattern: instrument everything. Every model call, every tool call, every policy decision, every user approval. Each one becomes a structured trace event with inputs, outputs, duration, and cost.

trace_event = {
  timestamp, trace_id, span_id, parent_span_id,
  kind: "model" | "tool" | "policy" | "approval",
  name: "tool:edit_file",
  input_summary: "path=src/auth.ts, remove GET bypass",
  output_summary: "Edit applied: 1 line changed",
  metrics: { duration_ms: 8, cost_usd: 0.013, tokens_in: 3800 }
}

Trace IDs tie events together within a session. Span IDs create parent-child relationships. When a supervisor dispatches a specialist, the specialist's events link back to the supervisor's span. Query by trace ID to see the full picture.

Two-tier storage

Trace events contain summaries. Raw artifacts — full prompts, full responses, complete stdout, file snapshots — go in a separate store. This keeps the trace index fast and queryable while preserving everything you need for replay.

When something breaks: query the trace, find the suspicious event, pull the raw artifacts, and reconstruct exactly what the model saw and did. No guessing.

Cost tracking changes behavior

The most underrated benefit of tracing: cost visibility. When you can see that a session cost $0.03 versus $4.80, and drill down to which tool calls consumed the most tokens, you make different design decisions.

I discovered that one of my agents was re-reading a 2,000-line file every iteration because my context assembler wasn't caching properly. That single bug was costing $0.50 per session in unnecessary token usage. Without cost tracking, I would never have found it.

The anti-pattern: tracing only model calls

Logging model calls without tool calls is like recording only one side of a phone conversation. You know what the model said, but not what actually happened when its instructions were executed. The model said "edit applied" — but did the edit actually succeed? Did it change the right lines? The tool trace tells you.

The takeaway

Tracing is the foundation for everything else: debugging, cost control, regression testing, incident response. Build it first. Instrument everything. Store the raw artifacts. You'll need them exactly once — and when you do, nothing else will substitute.

This is the final post in the "How Agents Actually Work" series. Start from the beginning: How Agents Act: The Tool Gateway