Prompt Caching vs KV Cache
The KV cache stores attention keys and values for the tokens already processed within a single request, so each new token does not reprocess the whole sequence. Prompt caching keeps that processed state for a shared prompt prefix across requests, so later calls that begin with exactly the same content are cheaper and faster.
Definitions
- Prompt Caching PTL-0081
- Prompt caching stores the model's processed state for a reused prompt prefix, such as a long system prompt or document, so later requests sharing that prefix are cheaper and faster.
- KV Cache PTL-0182
- The KV cache stores the attention keys and values a Transformer has already computed for earlier tokens, so that generating each new token requires computing only that token's keys and values instead of reprocessing the whole sequence.
Differences
| Prompt Caching | KV Cache | |
|---|---|---|
| Scope | Across requests | Within one request |
| Who controls it | The developer, by structuring prompts and marking breakpoints | The model server, automatically |
| Requirement | An identical prompt prefix | None |
| Benefit | Lower price and latency for repeated context | Fast token-by-token generation |
Which to use
- Use prompt caching by putting stable content such as system prompts and documents first.
- The KV cache is always in use; it matters when sizing memory for long contexts and many concurrent requests.
Sources
- Anthropic (2024). Prompt caching.
- Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention.
- Anthropic (2024). Prompt caching.
Cite this page
Protologue. (2026). Prompt Caching vs KV Cache. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0). https://protologue.com/compare/prompt-caching-vs-kv-cache/