Agent Context Windows: The Throughput Ceiling Nobody Budgets For

Context window exhaustion is the production failure mode that doesn't show up in your demo, doesn't trigger an alert, and quietly makes your multi-agent system far more expensive than your spreadsheet predicted.

Most teams budget for agent count and model tier. Almost nobody budgets for context pressure — the cumulative token load each agent carries across a multi-turn, multi-step workflow. By the time you notice, you're either hitting max_tokens errors mid-task, silently truncating critical state, or paying for context you refill on every retry.

What "Context Window" Actually Means in an Agent Loop

In a single-turn prompt, the context window is trivial to reason about. In an agent loop, it compounds. Every tool call appends its result. Every sub-agent response comes back as a message. Every retry re-attaches the prior turn. A workflow that takes 8 tool calls with moderately verbose outputs — think API responses, retrieved document chunks, structured JSON — can easily consume 40–60k tokens per task instance before the orchestrator even writes a final response.

Multiply that by concurrent agents and you're not dealing with a token-per-query problem. You're dealing with a working memory throughput problem.

Here's the math that actually matters:

code
context_load_per_task = system_prompt
                      + tool_schemas
                      + conversation_history
                      + sum(tool_call_results)
                      + in_flight_sub_agent_outputs

For a GPT-4-class model with a 128k context, that looks spacious until you realize your tool schemas alone are 3–5k tokens, your system prompt is another 2–4k (if you're doing it right), and a single RAG retrieval returning five chunks at 500 tokens each adds 2,500 tokens. Three retrieval steps and two code execution results later, you're at 30k tokens and you haven't produced a single output token yet.

The Three Context Pressure Patterns That Break Production

1. History accumulation without pruning

The easiest agent loop to write appends every message to a growing list and sends the whole thing on every turn. This is fine for demos. In production, after 10–15 turns, you're spending more on prompt tokens than on the model's actual reasoning. The fix is not truncation — naive truncation destroys task coherence. The fix is structured summarization at checkpoints: after N turns or after a logical phase boundary, compress completed work into a compact state object and drop the raw history.

python
# Bad: unbounded history
messages = conversation_history  # grows without limit

# Better: checkpoint compression
if len(messages) > CHECKPOINT_THRESHOLD:
    summary = summarize_completed_phase(messages[:-RETAIN_LAST_N])
    messages = [SystemMessage(content=summary)] + messages[-RETAIN_LAST_N:]

The tradeoff: summarization costs a model call. Budget for it. It's almost always cheaper than the alternative.

2. Tool result verbosity

Most tool call results are returned raw. A database query that could return a 10-row summary instead returns 200 rows of JSON. An HTTP response returns the full body when you needed three fields. Every verbose result pushes other context out — or runs you up against limits.

The rule I apply: every tool should have a detail_level parameter, defaulting to summary. Full results are opt-in when the agent explicitly needs them. In data-heavy workflows, this single change can cut tool result token load substantially — the gains depend on how verbose your tool responses currently are, but the pattern consistently moves the needle.

3. Sub-agent output re-ingestion

In a multi-agent system, the orchestrator often re-ingests the full output of sub-agents to decide what to do next. If sub-agent A returns a 4,000-token analysis, and sub-agent B returns another 3,000 tokens, and the orchestrator has its own 8k system prompt — the orchestrator's context fills up fast, and it hasn't even called its own tools yet.

The fix: sub-agents should return structured conclusions, not full workings. The detailed reasoning stays in sub-agent memory (or a shared scratchpad); the orchestrator gets a 200-token summary with a confidence flag and a result object. Think of it as inter-process communication — you don't pass entire stack traces between services, you pass return codes and payloads.

Context Window Sizing Is a Cost Decision, Not Just a Capability Decision

There's a seductive move that teams make when they hit context pressure: upgrade to a model with a larger context window. That's sometimes right. More often it's an expensive way to avoid fixing the underlying architecture.

Consider the cost structure. Models with larger context windows typically carry higher per-token pricing — and at 100k+ token contexts, even small per-token differences compound into real budget variance at scale. If your workflow is running 1,000 task instances a day and you're carrying tens of thousands of unnecessary context tokens on each, you're not paying for intelligence. You're paying for a bad data structure.

The routing decision I use:

Situation	Action
Context load < 30k tokens, stable	Stay on current model tier
Context load 30–80k tokens	Audit and prune before upsizing
Context load > 80k tokens consistently	Profile by component, compress aggressively
Context load > 80k and can't compress	Decompose the task, not the model

Decomposing the task — breaking a single large-context agent into a pipeline of smaller-context agents — is almost always the better architectural move. It also makes agent retry logic cheaper, because retries replay smaller units.

The Throughput Ceiling You're Not Measuring

Here's the non-obvious part. Context window exhaustion doesn't just cause errors — it creates throughput ceilings that don't show up until you're under load.

LLM APIs rate-limit by tokens per minute (TPM), not by requests per minute alone. A system where each request consumes 80k tokens can process far fewer concurrent tasks than one where each request consumes 20k tokens — even if latency per request is similar. At scale, context efficiency is throughput capacity.

Imagine a team running 50 concurrent travel research agents, each carrying 70k tokens of context. At a 10M TPM limit, that's roughly 142 requests per minute theoretical max. Trim average context to 20k tokens and the same limit supports around 500 requests per minute — a 3.5× throughput gain without changing infrastructure. The arithmetic is straightforward: TPM ÷ tokens-per-request = max requests per minute.

This is why context window management belongs in your capacity planning, not just your cost model.

What to Measure Before You Optimize

Don't optimize blindly. First, instrument your agent loops to surface the actual distribution of context loads per task type.

python
import tiktoken

def log_context_usage(messages: list, model: str = "gpt-4o") -> dict:
    enc = tiktoken.encoding_for_model(model)
    breakdown = {
        "system": 0,
        "tool_results": 0,
        "assistant": 0,
        "user": 0,
    }
    role_map = {
        "system": "system",
        "tool": "tool_results",
        "assistant": "assistant",
        "user": "user",
    }
    for m in messages:
        role_key = role_map.get(m["role"], "user")
        breakdown[role_key] += len(enc.encode(m["content"]))
    breakdown["total"] = sum(breakdown.values())
    # emit to your observability stack
    return breakdown

Once you have the distribution, the top three consumers are almost always: tool results, retained conversation history, and repeated system prompt boilerplate. Fix them in that order.

A context audit isn't a one-time exercise. Tool schemas evolve. Retrieved documents get longer. New tool calls get added. Wire this telemetry into your agent observability pipeline permanently and alert when average context load drifts above a threshold — say, 70% of your target window.

The Architectural Principle Nobody States Explicitly

Context window space is working memory. You wouldn't leave 80% of your RAM occupied by stale data and expect good performance.

The teams that run multi-agent systems cheaply and reliably in production treat context management as a first-class engineering concern — on par with latency and retry handling. The teams that treat it as an afterthought discover the problem via a surprise invoice or a silent accuracy degradation. Research on long-context LLM behavior — including the well-documented "lost in the middle" findings — shows that retrieval accuracy degrades for content positioned far from the beginning or end of a long prompt. Don't assume the model gracefully handles everything it technically fits in its window.

What to Actually Do

1. Instrument now. Add token-count logging per role (system, tool, assistant) to every agent loop. You can't fix what you can't measure, and the data will surprise you.

2. Set a context budget per task type — not a limit, a budget. A research task might warrant 60k tokens; a classification task should never exceed 8k. Treat overruns as bugs, not capacity requests.

3. Apply tool verbosity controls. Add a format: "summary" | "full" parameter to every tool your agents call. Default to "summary". Audit which tools return the most tokens and fix those first.

4. Implement phase-boundary compression for any agent loop that runs more than 6–8 turns. Checkpoint, compress, continue. Don't rely on the model to handle extremely long histories gracefully — the evidence suggests it won't.

5. Add context load to your capacity planning model. TPM limits are real. If you're planning to scale concurrent agents, calculate your peak tokens-per-minute load at target concurrency and check it against your API tier limits before you hit them.

Context pressure is the throughput ceiling you didn't know you were building toward. Fix the architecture, not the model tier.