Streaming vs. Batch LLM Calls: The Latency-Cost Decision Table

Streaming an LLM response feels faster. It looks faster. But in most multi-agent architectures I've worked across — including the travel platform we're building at Etera AI, where agent-to-agent calls outnumber user-facing calls by a large margin — defaulting to streaming everywhere quietly doubles infrastructure cost and creates throughput ceilings teams don't notice until they're already at scale.

This is the decision nobody writes a framework for because it seems obvious: stream for users, batch for background work. The reality is messier. The right call depends on token volume, downstream dependencies, your inference provider's billing model, and whether the response is even human-facing. Get it wrong and you're either paying for TCP connection overhead on millions of background calls or you're making users stare at a spinner when they could have seen the first token in 200ms.

Here's how I think about it.

Why the Default-to-Streaming Habit Is Expensive

Streaming keeps a connection open for the full generation duration. At scale, that single fact has compounding consequences:

• Connection pool saturation. Each open stream occupies a worker thread or async handler. If you're running hundreds of concurrent background tasks, you're holding that many open connections for the full generation window — not just the roundtrip. On a multi-agent system with tight concurrency limits, this degrades badly.
• No effective batching. Providers like OpenAI and Anthropic both offer batch inference at a significant discount (OpenAI's Batch API is publicly documented at 50% of standard pricing). Streaming is incompatible with batch APIs by definition.
• Retry amplification. A failed stream mid-generation means you retry the entire generation. A batch call that fails retries a complete payload, but you have full control over idempotency keys and partial result recovery. I covered this dynamic in more detail in Agent Retry Logic: The Silent Cost Multiplier in Multi-Agent Systems.

The per-token price is identical whether you stream or not on most providers. The infrastructure cost — load balancer timeout configuration, connection concurrency limits, error recovery complexity — is not.

The Four Cases Where Streaming Actually Wins

At Etera AI, we've had to be deliberate about this because the platform runs multi-agent loops that mix user-facing responses with a lot of internal orchestration. The places where streaming genuinely earns its overhead:

1. Real-time user-facing output. Chat interfaces, copilot sidebars, voice transcription pipelines where the user is watching. Every 100ms of perceived latency reduction matters here. Time-to-first-token (TTFT) is the metric, not total generation time.
2. Early-exit parsing. If you're streaming a structured response and you can bail out early when you've parsed the fields you need — say, a status: "reject" that appears in the first few dozen tokens — you save real money by cancelling the connection. This requires your parser to handle partial JSON, which most teams don't wire up.
3. Long generations with downstream fan-out. If generation produces tokens that trigger parallel sub-agent calls, streaming lets you start the fan-out before generation finishes. You get pipeline overlap that a blocking batch call can't give you.
4. Speculative rendering. UI patterns where you render partial markdown and update in-place. Users perceive this as a dramatically faster product even when total wall-clock time is the same.

Outside these four cases, streaming is probably the wrong default.

The Decision Table

Use Case	Recommended Mode	Why
Chat / user-facing generation	Stream	TTFT dominates UX
Background classification	Batch	50% cost saving, no UX impact
Agent-to-agent tool calls	Non-streaming async	Reduces connection pressure
Document summarization at scale	Batch API	Latency SLA is hours, not seconds
Structured extraction (small payload)	Non-streaming sync	Simpler retry, no partial parse
Long doc + early exit possible	Stream with cancellation	Save tokens if condition met early
Agentic loop inner calls	Non-streaming	Batch where possible, reduce overhead
Voice / realtime	Stream (chunked audio)	Latency is the entire product

The rule of thumb: if a human isn't watching the tokens appear in real time, you probably shouldn't stream.

Batching: The 50% Discount Most Teams Leave on the Table

OpenAI's Batch API charges 50% of standard token prices for requests submitted asynchronously with up to 24-hour turnaround. Anthropic has a similar offering. For any workload that can tolerate that window — nightly data enrichment, offline classification, bulk document processing, evaluation pipelines — you're burning money if you're not using it.

The integration pattern is straightforward. Here's the OpenAI Batch API shape, which is worth understanding before you decide whether to build the polling layer:

python
# OpenAI Batch API — submit a JSONL file of requests
from openai import OpenAI
import json

client = OpenAI()

# Build the batch input file
requests = [
    {
        "custom_id": f"doc-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [{"role": "user", "content": doc_text}],
            "max_tokens": 512
        }
    }
    for i, doc_text in enumerate(documents)
]

with open("batch_input.jsonl", "w") as f:
    for r in requests:
        f.write(json.dumps(r) + "\n")

# Upload and submit
batch_file = client.files.create(
    file=open("batch_input.jsonl", "rb"),
    purpose="batch"
)

batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

print(f"Batch submitted: {batch.id}")

The operational overhead is a polling job and a results parser. On any serious production pipeline running background enrichment or eval jobs, the cost savings are large enough to justify that work many times over. The engineering is fast — a day or two; the friction is usually getting the team to reclassify workloads they've always treated as synchronous.

Latency Decomposition: Know What You're Actually Optimizing

Most teams treat "latency" as a single number. It isn't. For LLM calls, latency breaks into:

• TTFT (Time to First Token): Network roundtrip + prompt processing time. Dominated by prompt length and model size. Caching your system prompt aggressively is the highest-leverage lever here.
• Inter-token latency: Generation speed in tokens/second. Mostly model-and-provider-dependent. Smaller models are substantially faster here — the throughput difference between a frontier model and its mini/flash variant is significant and well-documented by providers.
• Total generation time: TTFT + (output tokens × per-token generation time). This is what determines connection hold time and what you pay for in infrastructure terms.

When you're debugging a latency complaint, the first question is: which component is the bottleneck? A slow TTFT with fast inter-token suggests a prompt caching miss or a prompt length problem. Slow inter-token latency on short prompts suggests you're on a congested model endpoint or using a model that's oversized for the task.

Instrument all three separately. Don't aggregate them into a single "p95 latency" metric or you'll spend weeks tuning the wrong thing.

Model Routing as a Cost-Latency Control Surface

The biggest lever in cost-latency optimization isn't streaming vs. batch. It's model selection per call. The price-performance gap between frontier models and their mini/flash/haiku variants is substantial — providers publish these numbers openly, and the difference on output tokens especially is large enough to reshape your unit economics at volume.

A routing layer that sends classification and extraction to a cheaper/faster model while reserving the frontier model for synthesis and generation can cut the cost of an agentic loop dramatically without touching quality on most tasks. At Etera AI, the multi-agent architecture makes this unavoidable — you can't run every sub-agent on the most capable model and stay within sensible cost bounds. The implementation is a few dozen lines of routing logic around a cost and capability matrix.

Decision rule:

• Extraction, classification, structured output → GPT-4o-mini / Gemini Flash / Haiku
• Reasoning, synthesis, long-form generation → GPT-4o / Claude Sonnet or Opus / Gemini Pro
• Simple yes/no routing decisions → the smallest model that's reliable for the task, or a regex

Teams that don't build this routing layer are leaving large efficiency gains on the table. It's not novel architecture — it's basic cost discipline.

The One Antipattern That Kills Both Metrics Simultaneously

Streaming a large response to an intermediate agent that then blocks on the full response before continuing. This is the worst of both worlds: you hold an open connection for the full generation window (streaming cost), but the downstream agent doesn't start until generation finishes (no latency benefit). I see this constantly in hastily assembled multi-agent pipelines — it's especially common when teams retrofit streaming onto architectures that weren't designed for it.

If you're streaming to a non-human consumer, you must have a concrete reason. That reason is either early-exit cancellation or downstream fan-out that starts before generation ends. If neither applies, drop to a regular async completion call.

Streaming to an agent that waits for EOF is just an expensive way to make a blocking call.

What to Actually Do

1. Audit your current streaming usage this week. Pull your API call logs and tag every streaming call by consumer type: user-facing, agent-to-agent, or background. You will almost certainly find a large proportion of streaming calls with no human watching — and no early-exit or fan-out logic to justify it.
2. Migrate background workloads to Batch API. Any pipeline with a latency SLA measured in hours, not seconds, qualifies. Start with your evaluation and enrichment jobs — these are the easiest wins.
3. Instrument TTFT, inter-token latency, and total generation time separately. Aggregated latency metrics lie. You need the decomposition to know what to fix.
4. Build a two-tier model routing layer. Extraction and classification go to the cheap-fast tier. Synthesis and generation go to the capable tier. Wire it behind a single llm_call() abstraction so the routing is invisible to the rest of your codebase.
5. If you keep streaming to intermediate agents, add early-exit cancellation. Parse the stream incrementally and cancel the HTTP connection when you have what you need. The provider stops generating; you stop paying.

The infrastructure bill for most LLM-heavy systems is a routing and call-pattern problem disguised as a model pricing problem. Fix the patterns first.