protologue

KV Cache

Also called key-value cache, KV caching.

The KV cache stores the attention keys and values a Transformer has already computed for earlier tokens, so that generating each new token requires computing only that token's keys and values instead of reprocessing the whole sequence.

Description

Caching makes generation fast, but its memory grows with context length and with the number of concurrent requests, which is a main constraint on long contexts and serving costs. Serving systems such as vLLM manage it in pages to reduce waste. Prompt caching extends the idea across requests by keeping the cache for a shared prompt prefix, which is why cached prompts must begin with identical content to get the discount.

Sources

  1. Kwon et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention.
  2. Anthropic (2024). Prompt caching.

Cite this entry

Protologue. (2026). KV Cache. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0182). https://protologue.com/t/kv-cache/

BibTeX
@misc{protologue_kv_cache,
  title = {KV Cache},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0182},
  url = {https://protologue.com/t/kv-cache/}
}

Markdown JSON