KV Cache (Key-Value Cache)

The KV cache (key-value cache) is the working memory of a transformer model during inference. It stores the key and value vectors of every token already read, so each new token can be produced without recomputing the full context. Without it, text generation would be quadratically expensive; with it, per-token cost grows only linearly. Memory still scales with context length, because every layer and attention head contributes two vector blocks per token — which is why KV-cache size directly determines how many parallel sessions a GPU serves, how fast long-context requests run, and what a token costs. Modern architectures compress the cache deliberately: multi-head latent attention brings DeepSeek V4.1 Flash down to roughly 890 bytes per token. Closely related is prefix caching, where identical system prompts and document starts are reused across requests. For teams running AI agents in production, the KV cache is not an internal detail but a practical lever for throughput, price, and stability — especially in agent loops with long, repeated contexts.