Prompt Caching: How It Actually Cuts Your LLM API Bill

If you're building on top of large language models through an API, prompt caching is one of the highest-leverage cost levers you have — and one of the most misunderstood. It does not mean storing the model's answers and handing back the same text next time. That's response caching, a different technique. Prompt caching is a feature of the model provider: it saves the processed state of the input you send, so that when your next request starts with the same content, the model skips re-reading it and you're billed a fraction of the price for that part. You still get a fresh, freshly generated answer every time. What you stop paying for is the model re-processing the same long preamble over and over.

That distinction matters, because the savings are large and the technique is nearly free to adopt — if you understand what actually gets cached.

How prompt caching works#

When a model receives your prompt, the first thing it does is read it — the "prefill" step — building an internal representation of every token before it writes a single word of output. For a short question that's cheap. For a prompt carrying a big system message, a set of tool definitions, a 40-page document, or a long conversation history, prefill is most of the work and most of the cost.

Here's the key fact: that internal representation is built left to right, and each token's state depends on every token before it. So the model can reuse its work only from the start of the prompt up to the first point where anything changes. Think of it like a bookmark in a book the model otherwise re-reads from page one on every call. Prompt caching drops a bookmark after the last unchanged page; next time, the model resumes from the bookmark instead of starting over. But the bookmark is only valid if every earlier page is word-for-word identical — change one token near the front and the bookmark is worthless from that point on.

This is why prompt caching is a prefix cache. The reused content has to sit at the beginning of your prompt and be byte-for-byte the same across requests. The variable part — the user's actual question — goes at the end.

Providers expose this in one of two ways:

• Explicit (Anthropic Claude): you mark where the stable prefix ends with a cache_control breakpoint in the request. You decide what to cache, and you can set several breakpoints for layered content.
• Automatic (OpenAI, Google Gemini's implicit mode): the provider detects a repeated prefix and caches it for you with no code change. You just have to keep the front of your prompt stable.

Either way there's a floor and a clock. Caching only starts above a minimum prefix size, and that floor is provider- and model-specific — below it, shorter prefixes silently don't cache. On Anthropic the minimum depends on the model: roughly 2,048 tokens on Sonnet 4.6 and about 4,096 tokens on Opus 4.8 and Haiku 4.5. OpenAI's automatic caching starts around 1,024 tokens. And the cache is short-lived: Anthropic's default lifetime is about 5 minutes, refreshed each time you hit it, with an opt-in 1-hour window; OpenAI and Google keep prefixes warm on their own schedules. Miss the window and the next request rebuilds the cache from scratch.

What it costs, and what it saves#

The economics have two sides:

• Cache reads (hits) are cheap. On Anthropic, a cached input token bills at roughly one-tenth of the normal input rate — about a 90% discount. In list terms — Anthropic's published rates as of mid-2026 — that's Claude Sonnet 4.6 input dropping from $3 to around $0.30 per million tokens, Haiku 4.5 from $1 to about $0.10, Opus 4.8 from $5 to about $0.50. OpenAI and DeepSeek land in the same territory (DeepSeek's cache hits are also about a tenth of list), and Google's context caching applies a comparable discount.
• Cache writes (misses) can cost a small premium. The first request that lays down the cache pays for those input tokens in full, and on Anthropic a bit more — 1.25× the input rate for the 5-minute cache, 2× for the 1-hour cache. Automatic caches (OpenAI) generally don't surcharge the write.

Two consequences fall out of that. First, prompt caching pays off through reuse: you eat a slightly higher write once, then bank the discount on every hit before the cache expires. Cache a prefix you only send once and you've spent more, not less. Second, the bigger and more repetitive your stable context, the bigger the win — which is exactly the shape of the workloads people run today:

• Agents and tool use — a long system prompt plus a fat block of tool definitions gets re-sent on every step of every run. Cache it once per session.
• Multi-turn chat — the conversation history grows each turn, but its earlier turns don't change, so the whole prefix caches and only the newest message is new.
• RAG and document Q&A — pin one large document in the prefix and ask twenty questions against it; the document is billed at the cache rate for nineteen of them.
• Coding assistants — the same repository context rides along with every request.

And it's not only cheaper. Skipping prefill on cached tokens cuts latency too — time-to-first-token drops sharply when the model doesn't have to re-read tens of thousands of tokens before it starts answering.

Where it breaks#

Prompt caching is unusually easy to think you're using while getting nothing back. The failure modes are specific:

• A moving prefix. Anything that changes near the front of the prompt busts the cache for everything after it. The classic own-goal is injecting a live timestamp, a request ID, or a "current date" line at the top of the system prompt — every call looks new. Nondeterministic ordering (JSON keys, or retrieved chunks in a different order each time) does the same. Keep the front stable; push volatile content to the end.
• Traffic too sparse for the TTL. With Anthropic's ~5-minute default lifetime, a low-traffic endpoint can expire its cache between requests and pay the write premium each time with nothing to amortize it against. Other providers keep prefixes warm longer, but the principle holds: caching rewards steady or bursty reuse, not the occasional lone call.
• Prefixes below the minimum. Every provider has a floor below which nothing caches, and on Anthropic it's model-dependent — roughly 2,048 tokens on Sonnet 4.6 and around 4,096 on Opus 4.8 and Haiku 4.5, versus about 1,024 for OpenAI's automatic caching. Below the floor there's nothing to cache and small prompts don't benefit — and the miss is silent: cache_creation_input_tokens stays 0, no error.
• Scope and invalidation. The cache is keyed to your account (and, increasingly, your workspace) and to the exact model and certain request settings — it isn't shared across accounts, and switching models or changing cached tool definitions invalidates it. That isolation is good for privacy; it also means you can't borrow someone else's warm cache.

One worry that does not apply here: staleness of the answer. Because the model still generates fresh output on every call, prompt caching can't serve you an out-of-date response the way response caching can. The "is the cached data still current?" problem belongs to caching the answer, not to caching the prompt.

Getting the most out of it#

• Order the prompt static-to-dynamic. This is the single biggest lever. Stable, reusable material first — system prompt, tools, documents, examples — and the user's variable input last. Structure beats everything else here.
• Use layered breakpoints where you can. Content changes at different rates: tool definitions rarely, the system prompt occasionally, retrieved documents per session, the user turn every time. On providers that allow multiple cache breakpoints (Anthropic supports up to four), place them at those boundaries so a change in one layer doesn't invalidate the layers above it.
• Keep prefixes byte-identical. No timestamps, random IDs, or reshuffled JSON near the front. Serialize the stable part deterministically.
• Reuse inside the window. Fire related requests against the same context while it's warm, and consider the 1-hour TTL when reuse is predictable enough to justify the higher write cost.
• Measure the hit rate. Don't assume — read it. Anthropic returns cache_creation_input_tokens and cache_read_input_tokens; OpenAI reports cached_tokens in usage. If reads aren't dwarfing writes, your prefix is moving somewhere you didn't expect.
• Stack it with response caching when it fits. For requests that are truly identical end to end, caching the final answer is cheaper still than regenerating it. The two techniques solve different problems and combine cleanly.

The bottom line#

Prompt caching is close to free to turn on, and for the long, repetitive context that agents, chat, RAG, and coding tools all run on, it takes a real bite out of the bill — up to about 90% off the cached portion of your input, plus a latency win from skipping the re-read. The catch is entirely in the details it rewards: a stable, front-loaded prefix, big enough to clear the provider's minimum, reused often enough to beat the clock. Get the prompt structure right and the discount is automatic. Get it wrong — a timestamp at the top, a prompt reshuffled every call — and you'll pay the write premium forever and wonder why the cache never hits.