Prompt Caching Is the Fastest ROI in Your LLM Stack Right Now

Most production LLM deployments re-send the same system prompt, tool schemas, and retrieved documents on every single call. If your system prompt is 2,000 tokens and you're running 100,000 calls a day, you're paying for 200 million tokens that carry zero new information. Prompt caching is the fix — and most teams haven't touched it.

This isn't a research paper trick. Anthropic's prompt caching is live in production on Claude 3.5 and Claude 3 Haiku, OpenAI has context caching on GPT-4o, and Google has it on Gemini 1.5. The mechanics differ, the tradeoffs are real, and the implementation is non-trivial enough that most teams get it wrong the first time. Let's fix that.

What Prompt Caching Actually Does (and Doesn't Do)

The core idea: the inference server processes your input tokens once, stores the computed KV (key-value) cache — the intermediate attention representations — and reuses that state on subsequent calls with the same prefix. You pay a higher cost to write the cache on the first call, then a deeply discounted rate to read from it on subsequent calls.

On Anthropic's Claude:

• Cache write: ~25% more expensive than standard input tokens
• Cache read: ~90% cheaper than standard input tokens
• Cache TTL: 5 minutes (resets on each cache hit)

On OpenAI (GPT-4o and GPT-4o-mini):

• Caching is automatic for prompts ≥ 1,024 tokens
• Cache reads are billed at 50% of standard input price
• No explicit API control — OpenAI handles it server-side

On Google Gemini 1.5:

• Called "context caching", explicit API control
• Minimum 32,768 tokens to cache (yes, that's the floor)
• Storage cost per token-hour applies

The implications of these differences are significant. Anthropic gives you explicit control with a cache_control marker — you decide exactly where the cache boundary sits. OpenAI automates it but you can't tune it. Google's 32K floor means it's irrelevant for anything under a substantial document corpus. Pick your provider based on your actual use case, not brand preference.

The Three Cache Architecture Patterns That Actually Work

Pattern 1: Static system prompt caching

The simplest win. Your system prompt — role definition, rules, output format, guardrails — is identical across every user request. Mark it cacheable and you're done. The catch: your prompt must be genuinely static. Any templated variable ({user_name}, {current_date}) that sits before the cache marker invalidates the cache on every call. A very common mistake is putting a timestamp at the top of the system prompt for logging purposes, then wondering why cache hits are zero.

python
# Anthropic Claude — explicit cache control
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": STATIC_SYSTEM_PROMPT,  # must be truly static
                "cache_control": {"type": "ephemeral"}
            },
            {
                "type": "text",
                "text": user_query  # dynamic part, not cached
            }
        ]
    }
]

Move all dynamic content — timestamps, user IDs, session context — after the cache marker. The model processes the cached prefix and picks up from there.

Pattern 2: Tool schema caching

If you're running agents with tool calling, your tool definitions are typically 1,000–5,000 tokens of JSON schema that's identical across calls. This is free money. Cache the tool block. For agents with 20+ tools (which is where tool design starts to break in other ways), the savings on schema tokens alone can materially reduce per-call cost.

Pattern 3: Document/RAG context caching

This is the most powerful pattern and the trickiest. If you're doing retrieval-augmented generation where the same large document (a policy document, a product catalog, a legal brief) gets retrieved repeatedly, you can cache the document and only vary the question. The challenge: your retrieval system must return the exact same byte sequence for the cache to hit. Even a minor difference in chunking, whitespace normalization, or retrieval ordering breaks the cache.

Design for cache stability: normalize your retrieved chunks before insertion, sort deterministically, and consider pinning high-frequency documents as permanent cache candidates rather than relying on dynamic retrieval.

The Four Ways Teams Break Their Cache Hit Rate

1. Dynamic content before the cache marker Any variable that changes per-request and sits before cache_control kills the cache. Common culprits: request IDs prepended for traceability, datetime.now() in the prompt, user-specific personalization injected at the top. Audit your prompt assembly code before anything else.

2. Ignoring the TTL cliff Anthropic's 5-minute TTL means low-traffic applications get near-zero cache benefit. If your app handles 3 requests per minute, the cache expires between most calls. The cache is designed for high-throughput, concurrent workloads. For batch jobs or low-frequency scheduled agents, the write cost adds up with minimal read recovery.

3. Treating cache metrics as an afterthought You cannot optimize what you don't measure. Both Anthropic and OpenAI return cache hit/miss data in the API response — cache_creation_input_tokens and cache_read_input_tokens on Anthropic, cached_tokens under prompt_tokens_details on OpenAI. If these aren't flowing into your observability stack, you're flying blind. Wire them up before you do anything else. If your telemetry setup is still incomplete, this problem compounds across your entire cost picture — I covered the broader observability gap here.

4. Caching the wrong content Caching a 500-token system prompt that costs a fraction of a cent per cache write nets you almost nothing. The economics only work when you're caching substantial, frequently-reused context: multi-thousand-token system prompts, full tool schema libraries, or document contexts. Run the math before you instrument anything.

The Caching ROI Calculator

Don't eyeball this. Here's a simple decision framework:

Variable	Your value
Cacheable tokens per call	C
Daily call volume	N
Cache hit rate (estimate)	H
Standard input price (per 1M tokens)	P
Cache read price (per 1M tokens)	P_read
Cache write price (per 1M tokens)	P_write

Daily cost without caching: (C × N × P) / 1,000,000

Daily cost with caching: (C × N_miss × P_write + C × N_hit × P_read) / 1,000,000

Where N_miss = N × (1 - H) and N_hit = N × H.

For a directional illustration using approximate Claude 3.5 Sonnet pricing (check current rates at Anthropic's pricing page — these move): assume ~$3/M standard input, ~$3.75/M cache write, ~$0.30/M cache read, with 4,000 cacheable tokens, 50,000 daily calls, and 70% hit rate:

• Without caching: roughly $600/day
• With caching: roughly $155/day
• Net saving: directionally large, consistent across high-volume workloads

Run your own numbers with current rates. The savings are real at scale; at low volume, the math is marginal.

When to Skip Caching Entirely

Caching is not universally correct. Skip it when:

• Your traffic is below ~10 requests/minute and you're on Anthropic — the TTL will expire before you accumulate enough cache reads to recover the write cost.
• Your prompt is highly personalized per user — if more than 30% of your prompt content changes per call, your cacheable prefix shrinks to the point where the gain is negligible.
• You're on Gemini and your context is under 32K tokens — the floor makes it inapplicable for typical use cases.
• Your latency SLA is extremely tight — first-call cache writes can add latency. For applications where P99 latency on the first user request matters, the write penalty needs to be benchmarked, not assumed away.

Caching optimizes the steady state. Cold starts, low traffic, and highly variable prompts all degrade the return.

What to Actually Do

1. Audit your prompt structure today. Pull five recent API request logs and identify everything that's identical across calls. That's your cacheable surface area. Measure it in tokens.

2. Move all dynamic content to the end of the prompt. Restructure your prompt assembly so the static prefix — system instructions, tool schemas, reference documents — always comes first and is byte-for-byte identical.

3. Instrument cache metrics before you flip the switch. Add cache_read_input_tokens and cache_creation_input_tokens to your cost tracking dashboard. You need a baseline and a post-deployment comparison, not a gut feel.

4. Run the ROI calc for your actual traffic profile. Plug in your real numbers using current provider pricing. If the daily saving is under $20 at current volume, fix other leaks first — prompt verbosity, unnecessary tool calls, model tier mismatches — and revisit caching when volume grows.

5. Test cache stability in staging. Replay the same 100 requests and verify cache hit rate in the API response. If you're not hitting above 60% in a controlled test, your prompt assembly has a non-determinism bug. Find it before you go to production.

Prompt caching is one of the few LLM optimizations that requires zero model quality tradeoff. You're not compressing, quantizing, or truncating — you're just not paying to recompute the same thing twice. That's not clever engineering. That's table stakes.