Why Context Is Everything in Agent Design

By iteration 5, my agent was hallucinating. Not dramatically — it wasn't inventing files. It was subtler: it "remembered" a function signature from turn 1, but I'd edited that function in turn 3. The agent was working with a stale mental model, and every action it took based on that model made things worse.

The problem wasn't the model. It was what I was feeding it.

Lossy context, lossless storage

The pattern that fixed this: give the model the minimum it needs to act correctly, but store everything for replay. Two tiers of storage, two different purposes.

The model gets summaries, excerpts, compressed observations — all fitted within a token budget. The trace store gets raw file contents, full stdout, complete diffs. Lossy for the model. Lossless for debugging.

The context pipeline

Every turn, my context assembler builds a package:

budget = MAX_TOKENS - system_prompt - session_memory - user_message

1. Repo map (~800 tokens, always included)
   File tree + key symbols. Orientation.

2. Active file excerpts (fresh read, not cached)
   Only the lines relevant to the current subtask.

3. Recent observations (compressed)
   Last 3 turns, summarized.

4. Symbol search results (if budget allows)
   Top 5 matches for current task keywords.

If over budget, drop in reverse priority:
  symbols first, then old observations, then excerpts.
  Repo map always stays.

The key insight: active files are always freshly read. If the model edited a file two turns ago, it gets the current version, not its memory of what it changed. This single rule eliminated an entire class of bugs where the model worked with stale state.

Observation compression matters more than you think

A test runner produces 500 lines of output. Feed that raw into the next prompt and two things happen: the context window fills up with irrelevant stack traces, and the model fixates on noise instead of the actual failure.

Compressed: "3 tests passed, 1 failed. Failure: auth.test.ts line 24 — expected 401, got 200. Error in validateToken()."

That's 30 tokens instead of 2,000, and the model can actually reason about it.

The anti-pattern: full files every turn

I've seen agent setups that paste entire files into every prompt. By iteration 3, the context is 80% stale file content the model already read. Attention is diluted. Responses degrade. The model starts ignoring parts of its context because there's too much of it.

The fix: excerpts, summaries, and a token budget. Treat context window space like memory — it's scarce and everything in it should earn its place.

The takeaway

Your agent is only as good as what it can see. Engineer the context window as carefully as you engineer the code. Budget it. Compress it. And never trust the model's memory — re-read what matters.

Next: The Supervisor Pattern: Coordinating Multiple Agents