Learn · Beginner

Prompt Caching: Why AI Agents Pay Once to Read, Then Read for Pennies

Prompt caching is a way for an AI provider to store the processed form of a repeated chunk of text -- most often a long system prompt or a big reference document -- so that it can be reused across many requests at a fraction of the normal cost, instead of being re-read from scratch every time. It is one of the biggest levers on what an AI agent actually costs, and it is invisible until you measure it -- which is exactly what a wire-level teardown of two coding agents did this week, finding one agent quietly rewriting tens of thousands of cache tokens mid-session.

To see why caching matters, you need one fact about how language models work. A model has no memory between turns; on every request it re-reads the entire input -- your system prompt, the tool definitions, the whole conversation so far -- and processes all of it before producing a single new word. For an AI agent, that input is dominated by a large, unchanging opening chunk: instructions and tool schemas that can run tens of thousands of tokens. Without caching, that chunk is re-processed on every single turn. Processing text is the expensive part, so you are paying full price, over and over, to re-read something that never changed.

Prompt caching fixes this. The first time the provider sees that opening chunk, it does the expensive processing and stores the result -- specifically the internal key-value representations the model computes, closely related to the KV-cache used within a single generation. On later requests that begin with the identical chunk, the provider skips the work and reads the stored version back. Providers price this steeply in your favor: a cache read typically costs a small fraction of processing the same tokens fresh. Anthropic, for example, charges a one-time premium to write to the cache and then charges roughly a tenth of the normal rate to read from it.

Here is the analogy. Imagine a lawyer who must re-read a 50-page contract before every phone call with you. If they re-read all 50 pages each time, you pay for 50 pages of reading on every call. Prompt caching is the lawyer reading it once, taking detailed notes, and then just glancing at the notes on future calls -- charging you a small fee to glance instead of a full fee to re-read. The catch is that the notes are only valid if the contract has not changed. Change a single clause and the lawyer has to re-read the whole thing.

That catch is the crucial, cost-defining detail: prompt caching is prefix-based and byte-exact. The cache matches from the very start of the input, and it only works up to the first point where the text differs from what was stored. If your opening chunk is byte-for-byte identical every time, you get the discount on the whole thing. But if anything near the top changes -- a timestamp injected into the system prompt, a reordered tool list, a rewritten instruction block -- the match breaks at that point, and everything from there on must be re-processed at full price. This is why a well-designed agent keeps its prefix stable (put the volatile stuff at the end, not the top) and why the coding-agent teardown found such large cost gaps: an agent that rewrote its own prefix mid-session was paying the full premium again and again for content it had already sent, buying no extra quality at all.

Why this matters beyond cost: caching also cuts latency, because skipping the re-processing of a huge prefix means the model starts responding sooner. And it reshapes how you should structure prompts and context windows. The practical rules fall right out of the mechanism: keep the stable, reusable material (instructions, tool definitions, long documents) at the front where it can be cached, and keep the changing material (the user's latest message, fresh retrieved data) at the back. For agents that make many turns over a long session, the difference between a stable cached prefix and a churning one can be a 10x or worse swing in cost for identical work.

The honest caveat: caching is not free or permanent. There is a write premium the first time, cache entries expire after a period of inactivity (so a cold cache pays full price again), and the byte-exact matching means small, well-intentioned changes can silently destroy the discount. Caching also does nothing for the parts of a request that genuinely change every turn. It is a powerful optimization for the repeated skeleton of your requests -- and understanding it turns an invisible, mysterious bill into something you can actually control. The foundational research on reusing attention computation across prompts is the Prompt Cache work, built on the attention mechanism at the heart of every modern model.

Ground Truth. (2026, July 12). Prompt Caching: Why AI Agents Pay Once to Read, Then Read for Pennies. Ground Truth. https://groundtruth.day/learn/prompt-caching.html

@misc{groundtruth:prompt-caching,
  title  = {Prompt Caching: Why AI Agents Pay Once to Read, Then Read for Pennies},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/prompt-caching.html}
}

Topics: prompt-caching · kv-cache · inference · cost · agents