When KV Cache Becomes an AI Inference Memory Bottleneck
The short answer
A KV-cache bottleneck appears when an inference service cannot keep the attention state needed by its active requests in the memory budget available to the serving engine. Long prompts, long generations and more simultaneous requests all increase that state. The service may then queue work, preempt requests, recompute state, offload blocks or reject traffic. The exact behavior depends on the model and runtime. A rising context window alone does not prove that KV capacity is the limiting factor: model execution, memory bandwidth, batching and the network can also set the pace.
This article explains what the cache does, how the pressure shows up in measurements and how to compare remedies. For the arithmetic and an RFQ-ready worksheet, use the inference memory sizing guide. Its illustrative numbers are planning examples, not TORmem benchmark results.
The discussion focuses on autoregressive language-model serving, where a request builds attention state as tokens are processed and generated. Image, audio, video and hybrid models can add encoders or state with a different shape, so their memory use needs a separate model-specific check. The practical method still applies: identify the live state, measure it under representative traffic, and compare complete services at the same quality and latency target. That prevents a simple “bytes per token” estimate from becoming an unsupported claim about a production workload.
Where KV state appears in a request
A decoder-based language model handles a request in two broad phases. During prefill, it processes the incoming prompt and creates attention state for those tokens. During decode, it produces new tokens, usually one step at a time. At each step, attention needs information from earlier tokens. The cache keeps the prior key and value tensors so the engine can reuse that state instead of recalculating it from the beginning for every generated token.
Keeping state is a tradeoff. An active request occupies cache capacity for its prompt and generated continuation. Longer output keeps that request resident longer; greater concurrency puts more requests in the cache at once. The model architecture changes the bytes per token: layer count, number of KV heads, head dimension and cache format all matter. Grouped-query attention can use fewer KV heads than query heads. Hybrid or sliding-window models may retain different state across layers. This is why a parameter count or advertised maximum context length cannot by itself determine an inference system's memory requirement.
The PagedAttention paper describes why managing this growing state efficiently is important to serving throughput. Its implementation divides the KV cache into blocks to reduce waste and make sharing possible. A block-based allocator improves use of available capacity; it does not remove the need to hold the live working set somewhere.
Why GPU memory becomes crowded
GPU high-bandwidth memory must hold more than the KV cache. Model weights, runtime workspace, graph captures and other allocations compete for the same device budget. The engine may reserve a cache pool after loading the model, or accept an explicit cache-size setting. The usable capacity is therefore specific to the loaded model, GPU configuration, engine version and parallelism mode. The sum of HBM across every GPU in a cluster is not necessarily a single pool that one request can use.
Picture three traffic shapes using the same model. A short-answer assistant with modest concurrency may have ample cache capacity and be limited elsewhere. A document assistant may see long prompts with a repeated system prefix; its opportunity may be prefix reuse as much as raw capacity. A service holding many long-running agent sessions can accumulate a large live set even if each incoming prompt looks ordinary. The decision is made from the overlapping requests and their resident tokens, not from the largest context number on the model card.
| Traffic shape | Likely memory question | First measurement |
|---|---|---|
| Short answers | Is the cache actually full? | Occupancy, preemptions and latency at peak request rate |
| Long documents | Are shared prefixes reused or repeatedly prefilled? | Prefix hit rate and time to first token |
| Persistent sessions | How much state overlaps in time? | Active sequences, live tokens and queueing |
For an actual estimate, calculate bytes per token from the model and multiply by the live tokens in representative busy windows. Then reconcile the estimate with the engine's reported allocation. The vLLM engine arguments show how one widely used runtime controls its KV-cache budget. They also illustrate why a configuration setting must be interpreted per GPU and per engine version.
How to tell cache pressure from another bottleneck
Start by holding the model and server configuration fixed while changing the request mix. Record offered load, completed load, prompt length, generated length, concurrency and the cache allocated by the engine. Then place request latency beside cache behavior. Time to first token (TTFT) captures the wait through routing, queueing and prefill. Time per output token (TPOT) describes decode speed after the first token. Neither metric alone names a root cause, but their movement with occupancy, preemptions and errors narrows the search.
| Observed pattern | Possible explanation | Countercheck |
|---|---|---|
| Cache full; preemptions rise | Live KV demand exceeds the allocated pool | Compare active tokens and allocation; watch queueing and errors |
| TTFT rises with long prompts | Prefill work, scheduling or cache misses | Check prefix hits, queue time and prefill load |
| TPOT rises with spare cache | Decode compute, bandwidth or batching | Check GPU utilization and serving settings before adding capacity |
| Errors under a burst | Admission, memory or upstream limits | Separate out-of-memory events from timeouts and client cancellations |
The important comparison is completed work within a service target. A run can report high output tokens per second while making users wait too long for a first token. It can also show low average latency because difficult requests timed out and vanished from the average. Keep rejected, failed and timed-out requests in the result. vLLM's metric documentation describes request and cache signals that can help explain the result; available names vary with runtime version.
A decision order for relieving memory pressure
Once the measurements show a real capacity problem, compare changes from the least disruptive to the most architectural. Each changes a different part of the service. Keep the model, traffic and latency target fixed when evaluating candidates, and record any quality or operational cost alongside throughput.
- Control admission and routing. Set a documented policy for extremely long requests, bursts and sessions that remain idle. Queueing or routing a long-context class to its own pool can protect shorter requests. This changes user experience, so state the policy instead of quietly truncating or rejecting work.
- Use the cache efficiently. Verify block allocation, prefix caching and eviction behavior in the chosen engine. A repeated system prompt can save prefill work when its blocks remain eligible and resident. Low reuse or heavy contention changes the outcome. vLLM documents how prefix blocks are reused and evicted.
- Examine representation and model choice. A supported lower-byte KV format or a model with fewer KV heads may reduce payload. Validate answer quality and serving behavior before counting the theoretical saving. A model change also affects output quality, prompt compatibility and operational maintenance.
- Add local capacity or replicas. A GPU with more usable HBM can increase per-worker headroom. Additional replicas can divide traffic, though each replica may load its own weights and cache. The right choice depends on the maximum request a single worker must serve and the normal utilization of the fleet.
- Consider an additional memory tier. Host memory or another supported tier may retain colder KV blocks or reusable prefixes. This adds movement and recovery behavior to the design. Require the exact runtime connector, data placement policy, transfer path and miss-rate evidence before treating capacity as usable.
Separate KV offloading from disaggregated serving. Offloading moves cache blocks between memory tiers. Disaggregated serving separates prefill and decode workers and transfers KV state between them. The latter may help a workload with competing prefill and decode demands, but it is not automatically a solution to insufficient cache capacity. NVIDIA Dynamo describes the prefill/decode split and its transfer path.
When another memory tier is worth testing
Another tier is attractive when the service repeatedly needs more reusable state than fits economically in HBM, and its runtime can decide which data to keep near compute. The buyer should identify the intended win: longer contexts, more active requests, fewer repeated prefills or lower cost at the same service level. Those are distinct claims. Keeping cold prefixes in host memory may help a high-reuse workload; it does not necessarily increase the number of active decode requests that fit on a GPU.
A miss crosses a transfer path. Its cost depends on block size, hit rate, bandwidth, contention, scheduling and where the data lives. An external memory device therefore needs application integration, not only a fast fabric. Ask what happens if the tier or link is unavailable: can the engine recompute, fall back to local memory, retry elsewhere, or does the request fail? NVIDIA's native KV-offloading documentation provides one concrete example of the software and routing requirements.
The shared AI memory guide compares local memory, CXL and network-accessed approaches. They have different access paths and programming models. A generic promise of pooled terabytes does not show that an inference runtime can place its KV cache there, or that doing so meets a latency target. Any product-specific performance conclusion needs a disclosed configuration and repeatable workload test.
A buyer test that produces a useful answer
Give every candidate the same model revision, prompt and output-length distributions, request arrival pattern and service target. Run a typical mix, a long-context burst and a low-prefix-reuse case. Report time to first token, time per output token, tail latency, completed requests, errors and cache pressure together. If a proposed design adds a tier, include cache-hit rate and transfer volume. Repeat after a worker or tier fault when recovery is part of the purchase decision.
Begin with a baseline that uses the serving configuration already in production or the simplest configuration being considered. Record its software versions and measured behavior before changing cache size, precision or placement. For each variant, change one main variable and rerun the same trace at several arrival rates. This makes a result such as “more throughput” interpretable: it may come from admitting more requests, reducing repeated prefill, faster decode or simply relaxing latency. Keep a separate quality check if the model or cache representation changes.
Ask for both the measurement and the conditions around it: engine and driver versions, GPU and host configuration, memory formats, parallelism, admission policy and request trace. vLLM's serving benchmark is one way to record request latency and throughput. The exact tool matters less than a reproducible comparison at the same target. For an RFQ-ready capacity worksheet and test matrix, continue to the inference memory sizing guide.
Read capacity claims as an operating decision
A larger cache can be valuable even without a faster single request: it may let one worker serve more concurrent sessions at the same latency target. The reverse is also possible. An extra tier may store many gigabytes of state while transfers make a particular request class slower. Compare cost per completed request or output token within the agreed service target, including timeouts and retries. The bill should include GPUs, host memory, any additional appliance or fabric capacity, power, software integration and the operational work needed to keep the tier healthy.
Ask what happens during a high-load hour and during a fault. Does a failed tier trigger recomputation, a retry on another worker or loss of the request? Does the fallback still meet the service target, and can the operator see it happening? A design that needs perfect cache reuse or an unavailable future connector to achieve its advertised capacity has a different risk profile from one that passes the same test today. Write the dependency and recovery path into the acceptance plan before using a capacity figure in a purchase decision.
Frequently asked questions
- What does KV cache store?
- It stores the attention key and value tensors produced for tokens already processed by a decoder-based model. During generation, the model can reuse those tensors when attending to earlier context. The cache is state associated with a request or reusable prefix, not a copy of the model weights. Its size depends on architecture, cache format and the number of resident tokens. Serving engines may divide it into blocks, retain selected prefixes after a request completes and evict blocks under pressure. Confirm the exact behavior in the runtime you intend to deploy.
- Why does a long context consume more memory?
- More prompt tokens create more attention state, and generated tokens add to that state while a request remains active. The same model can therefore have a small live KV set for short questions and a much larger one for document analysis or lengthy agent sessions. Maximum context length describes what a model or runtime can accept under some conditions; it does not tell you how many such requests can run together. Size the overlapping live tokens at the traffic level you need to serve, then check the engine's actual allocated cache capacity.
- Does high GPU-memory use prove a KV-cache bottleneck?
- No. GPU memory also holds model weights and runtime allocations, and some engines reserve a large cache pool at startup. A high allocation can be normal even when request latency is limited elsewhere. Look for cache occupancy under load, preemptions, queueing, out-of-memory events and the relationship between those signals and TTFT or TPOT. If latency worsens while the cache has headroom, investigate prefill work, decode compute, batching, bandwidth and upstream delays. Change one condition at a time so the result identifies a cause.
- How are TTFT and TPOT different?
- Time to first token includes the path to the initial output: routing, queueing and prompt processing can all contribute. Time per output token measures the pace of subsequent generation. Long prompts may raise TTFT even with enough cache capacity, while a slow decode phase can raise TPOT without a cache shortage. Review both at the same request mix and offered load, including P95 and P99 values. Also count failures and timeouts; a system that drops difficult requests can look fast if only successful responses appear in the latency chart.
- Can prefix caching solve the memory problem?
- It helps when requests repeat an eligible prefix and its blocks remain resident. That can avoid some repeated prefill computation. It also occupies memory after earlier requests finish, so cache policy and eviction matter. A system prompt shared by many requests may offer reuse; a stream of unique documents may not. Measure actual prefix hits under the expected mix and under a low-reuse stress case. Keep tenant isolation and privacy requirements in the test, because they may restrict which requests may share cached state.
- Is KV-cache quantization an easy capacity win?
- A supported smaller cache format can reduce bytes per token, but the theoretical saving is only a starting point. Check that the model, engine and attention backend support the format, and measure response quality and latency after enabling it. Allocation granularity and scaling metadata can affect the physical footprint. Run the same prompt mix and quality checks as the baseline, especially for long contexts and tasks sensitive to small output changes. Put the exact format and software versions in the test report so the result can be reproduced.
- Does adding more GPUs fix a full KV cache?
- It can help if additional replicas divide the traffic or a different parallelism mode changes per-worker memory use. It may not help one very long request if that request still has to fit within one worker's supported configuration. Replicas also repeat some weight and runtime allocations, so aggregate HBM is not the same as freely shareable cache capacity. Ask for the placement of weights and KV state on each GPU and test the maximum request size alongside total throughput. The cost comparison should include the extra hardware, power and normal utilization.
- What is the difference between KV offloading and disaggregated serving?
- KV offloading moves selected cache blocks between memory tiers, such as GPU and host memory, so reusable state can exceed the GPU-resident cache. Disaggregated serving separates prompt prefill and token decode onto different workers and transfers KV state between those stages. A deployment can use either technique, both or neither. Each adds a different transfer and scheduling path. Evaluate the technique against the actual limit: cache capacity, prefill contention or decode latency. A topology diagram and workload test make the distinction concrete.
- When should we consider network-accessed memory?
- Consider it when measured reuse or working-set demand exceeds economical local capacity and the serving stack has a supported way to move the relevant data. Specify whether the tier stores cold reusable prefixes, blocks needed during active requests or another object entirely. Then measure hit rate, transfer volume, first-token latency, tail latency and failure behavior against a local-memory baseline. The network path, software policy and host integration are part of the product. A fabric bandwidth number or appliance capacity figure alone does not establish an inference benefit.
- What evidence should a buyer request before purchasing a memory tier?
- Request a per-worker memory budget, model and runtime versions, traffic trace, baseline configuration and the proposed placement policy. The test report should give completed work within an agreed latency target, errors, P95/P99 behavior, cache occupancy, hit rate and transfer volume. Ask who integrates the tier, monitors it and handles failure. If the proposal claims fewer GPUs or lower serving cost, require a side-by-side comparison at the same quality and service target, with the full BOM, power and operating assumptions visible.
