The KV Cache Tax: Why Inference Servers Run Out of Memory Before Compute

The Model Fits, the Requests Don’t

VRAM used to be a training-time worry. You sized your cluster for the weights, the optimizer states, and the gradients, and once the model was trained, the memory math felt settled. Serving looked cheap by comparison: load the weights, run forward passes, done.

Then you put the model behind real traffic, and it fell over at a load that made no sense.

I was sizing an inference deployment for a mid-sized model on a shared GPU pool. The weights fit with room to spare, and the server ran fine in testing. Under concurrent traffic, it returned CUDA out-of-memory errors while GPU compute utilization sat far below saturation.

The first instinct was that the model was too big. It wasn’t. The key-value cache was eating the headroom, and it grew with every concurrent request. I turned on paged allocation and prefix caching before adding a single GPU, and the ceiling lifted. Buying hardware would have papered over a memory-layout problem.

That is the tax most serving guides underprice. A vLLM analysis found that naive KV cache management wastes 60 to 80 percent of the memory it reserves. The cache is not a fixed cost you pay once. It scales with how many requests you serve at the same time, which means the thing that pushes you over the memory cliff is a traffic spike, not a longer prompt.

The VRAM Budget Equation: Why Memory Scales With Concurrency

Three things share GPU memory during serving: the model weights, transient activations, and the KV cache. The weights are fixed. The KV cache is the variable that moves.

The per-token size comes from four factors multiplied together:

• Two bytes each for the key and the value stored per token.

• Layers: every transformer layer keeps its own key/value pair.

• Key/value heads times head dimension: the width of each cached vector.

• Bytes per element: the precision you store the cache in.

Grouped-query attention shrinks the key/value head count, the term that matters most. That’s why modern architectures lean on it, and NVIDIA’s inference guide walks through the same arithmetic.

Put Llama 3.1 70B through it at BF16. With 80 layers, 8 key/value heads, and a head dimension of 128, each token costs roughly 320 KB of cache (2 x 80 x 8 x 128 x 2 bytes). Without grouped-query attention, at 64 heads, that figure would be eight times larger.

Now multiply by traffic. At 128 concurrent requests holding 4K tokens of context each, the KV cache alone needs about 160 GB. The 70B weights need 140 GB at BF16. The cache has outgrown the model. You can serve that model on a single 80 GB card until the moment concurrency climbs, and then no amount of spare compute helps, because the ceiling you hit is memory.

Here is the reframe. Your KV budget is a function of concurrent requests, not prompt length in isolation. vLLM sizes its cache as max concurrent sequences times max sequence length, so raising either one multiplies the reservation. Practitioners hit this as a cryptic startup error before a single request lands, because the server pre-allocates the whole cache up front.

The chart below shows the same GPU under rising concurrency: weights stay fixed while the cache climbs into the memory ceiling and triggers the OOM on the spike.

Naive Allocation: The 60 to 80 Percent You Never Use

Early serving stacks reserved KV memory in one contiguous block per request, sized for the maximum possible sequence length. A request that generated 200 tokens against a 4,096-token reservation left the rest parked and unusable.

The vLLM blog measured the damage: 60 to 80 percent of allocated KV memory wasted to fragmentation and over-reservation. That waste is the reason you run out of memory at low utilization. The GPU has the capacity; the allocator has fenced it off.

Think of operating-system virtual memory. Programs address memory as if it were one continuous space, while the OS maps those addresses to scattered physical pages behind the scenes. Nothing needs a giant contiguous block, so nothing gets stranded. The fix for the KV cache borrows exactly this idea.

PagedAttention: Virtual Memory for the KV Cache

PagedAttention stores the KV cache in fixed-size blocks that need not be contiguous, the same way an OS pages memory. Each request draws blocks as it generates tokens and returns them when it finishes, so reservations track actual usage instead of the worst case.

The payoff is direct. The vLLM team reports waste dropping to under 4 percent, and throughput rising up to 24 times over a naive Hugging Face pipeline because the reclaimed memory lets more requests batch together. More resident requests per GPU means higher throughput from the same hardware.

The cost is bookkeeping. Non-contiguous blocks mean the attention kernel has to gather scattered memory, and the block table adds a layer of indirection. In practice that overhead is small against the memory you win back. For any production serving stack in 2026, this is the baseline every deployment should start from.

Prefix Caching: Amortizing the Shared Prompt

Many workloads send the same tokens over and over. A shared system prompt, a few-shot preamble, a long document that every question in a session refers back to. Recomputing that prefix for each request burns compute and re-stores identical KV entries.

Prefix caching keeps the KV for a shared prefix once and reuses it across every request that starts with the same tokens. SGLang’s RadixAttention organizes cached prefixes in a radix tree so overlapping prompts share state automatically. The SGLang team reports up to 5 times higher throughput on workloads with heavy prefix reuse.

The benefit is conditional, and this is where teams misjudge it. Reuse only helps when requests actually share prefixes. On traffic without shared prefix structure the hit rate stays low, and the cache spends effort on lookups that mostly miss.

Before enabling it, track your cache hit rate; it’s the number that tells you whether the feature earns its keep. A chat product with a fixed system prompt will see high reuse. A service fielding unique one-off prompts won’t, and there the cache is overhead.

KV Quantization: Trading Precision for Capacity

If paging and prefix caching still leave you memory-constrained, store the cache in fewer bits. FP8 KV cache halves the per-token cost and roughly doubles the tokens you can hold.

Going further, the KIVI work quantizes to 2 bits for about 2.6 times less peak memory, enabling roughly 4 times larger batches and a 2.3 to 3.5 times throughput gain.

The cost is quality, and it isn’t uniform. For many tasks the drop is small. On long-context retrieval it can fall off a cliff. vLLM documented a needle-in-a-haystack score collapsing from 91 percent at BF16 to 13 percent under naive FP8, then recovering to 89 percent once they accumulated in higher precision.

The lesson: KV quantization is safe only when you validate it on your own long-context workload before you trust any aggregate benchmark.

It’s like saving photos at a lower resolution. Each one takes less space, most look fine, and the ones with fine detail are where you notice the loss. Long-context retrieval is that fine detail.

There is a further wrinkle worth knowing. Quality impact is architecture-specific: smaller reasoning models tend to degrade more than larger ones, and some attention variants break on uncalibrated scales. Treat the capacity gain as real and the quality as something you have to measure.

The Decision Matrix: Mapping Traffic to Strategy

Do not pick strategies from a diagram. Profile your traffic first: peak concurrency, prompt structure, context length, and how much your prompts actually overlap. The right stack follows from the traffic shape. The flow below maps the common branches.

Layer the strategies in this order:

• Start here: PagedAttention on every deployment, no exceptions.

• Add when prompts overlap: prefix caching, once you have confirmed a real hit rate.

• Add when still memory-bound: KV quantization, once you have validated long-context quality.

One caveat before you optimize. The “memory before compute” thesis is not universal. Some workloads become compute-bound before they exhaust KV memory, and past a workload-specific point, adding cache capacity buys nothing. Measure which wall you hit first. If your p95 latency is climbing while memory sits idle, more KV headroom is the wrong fix.

Conclusion

Turn on PagedAttention and prefix caching on day one. Paging is table stakes, and prefix caching is close to free when your traffic has shared structure. Together they solve the memory pressure most serving workloads actually have.

Reach for KV quantization only when you’re still capacity-constrained after those two, and only after you validate quality on your own long-context traffic. Size your deployment around peak concurrency, not average load, because the OOM arrives on the spike. The model fitting in memory was never the question that mattered.

Further Reading

• Efficient Memory Management for LLM Serving with PagedAttention (SOSP 2023, the founding vLLM paper on KV cache paging)

• Mastering LLM Techniques: Inference Optimization (NVIDIA’s reference on KV math, GQA, and MQA)

• SGLang and RadixAttention (prefix reuse via a radix tree)

• KIVI: Plug-and-play 2bit KV Cache Quantization (low-bit KV quantization and its throughput gains)

• FP8 KV Cache Accuracy (vLLM’s long-context regression and the fix)

• KV Cache Calculations (per-token worked examples across model families)

···

Thanks for reading. I’m Mostafa Ibrahim, founder of Codecontent, a developer-first technical content agency. I write about agentic systems, RAG, and production AI. If you’d like to stay in touch or discuss the ideas in this article, you can find me on LinkedIn here.