How much memory does an LLM inference workload need?

Collect the inputs before choosing GPUs

A model name and its maximum context window are insufficient for capacity planning. Two services using the same model can have very different memory demand if one serves short prompts at low concurrency and the other holds many long conversations open.

Gather a representative request trace if you have one. If you do not, use a few explicit scenarios—typical, busy hour and unusually long context—and label them as assumptions. The KV-cache overview explains why this working set matters.

Keep prompts and completions separate in the input sheet. Prompt tokens enter during prefill; output tokens arrive during decode and keep the request active longer. A service with 2,000-token prompts and brief answers has a different occupancy pattern from one with 500-token prompts and long generated responses, even if their total tokens per request are similar. Also record whether clients stream, disconnect early or keep sessions warm for follow-up requests.

Estimate the KV-cache payload

For a conventional transformer storing keys and values at each attention layer, an initial per-token estimate is 2 × layers × KV heads × head dimension × bytes per KV element. The factor of two accounts for keys and values. Multiply by the sum of active prompt and generated tokens across requests to estimate the live payload. Use the model’s KV-head count, not its query-head count.

Repeat the calculation for the busy-hour mix, including the tokens expected to be generated before a request finishes. If some requests share a reusable prefix, calculate a second scenario with observed cache reuse; do not assume every duplicate prompt will be retained. The serving engine may allocate pages or blocks, round up incomplete blocks, retain cached prefixes and reserve extra capacity. Some model architectures also need a different calculation. Compare the worksheet with the engine’s reported capacity.

vLLM documents its block-based prefix cache and its cache memory controls. Those controls are useful for checking the estimate against an actual deployment; they are not a substitute for measuring the request mix.

Convert traffic into simultaneously resident tokens

The sizing unit is the number of tokens whose KV data must be available at the same time. Requests per second alone does not give that number. You need the number of requests active concurrently, their current context lengths, how quickly they finish and whether completed prefixes remain cached. A traffic trace with timestamps and token counts is more useful than a single average prompt length.

The table uses the hypothetical model in the worked example. Its purpose is to show how the same model can demand very different cache capacity when context length and concurrency change together. The values are raw KV payload estimates. They exclude model weights, block rounding, runtime allocations and retained prefixes; they are not a benchmark or hardware recommendation.

1. Group real or expected requests by prompt length and output length. Preserve the long tail instead of replacing it with an average.
2. For each time window, add the live tokens of requests that overlap. Use the busiest representative windows to form the planning case.
3. Calculate a second case for a long-context burst or an unusually high arrival rate, and decide whether those requests must start immediately.
4. If prefix reuse is expected, keep a separate retained-cache budget. Measure whether the engine actually preserves the shared blocks under the chosen load.

This separation helps the buyer set an admission policy. A system may serve its normal mix comfortably while a few very long requests consume the cache needed by many short ones. Queueing, routing long requests to another pool or setting a context limit are product decisions as much as hardware decisions. Put the intended behavior in the service brief.

Check where the simple formula stops working

The per-token expression is useful for a conventional attention cache. It is not a universal formula for every model or runtime. Read the actual model configuration and ask the serving team which tensors are stored, in what format and on which device. A quoted parameter count or marketing context length does not answer those questions.

• Grouped-query or multi-query attention: fewer KV heads than query heads reduce the cache payload relative to an estimate based on all query heads.
• Hybrid or sliding-window attention: different layers may retain different amounts of state; multiplying one layer shape by every layer can mislead.
• Quantized KV: smaller elements can reduce payload, but the supported format, scaling metadata, model quality and runtime behavior need verification.
• Multimodal requests: images, audio or video can introduce additional tokens, encoders and intermediate state that a text-only worksheet misses.
• Partitioned serving: the cache and weights may be split or replicated across devices according to the engine and parallelism mode.

Use the formula to catch order-of-magnitude mistakes and compare traffic scenarios. Use the engine’s allocation report to determine the deployable configuration. If the two disagree, investigate the architecture and allocator before buying more memory; the difference may be a modeling error, a configuration limit or a real runtime overhead.

Build the full memory budget

KV cache competes with model weights, execution workspace and the serving runtime for device memory. Weight memory can be approximated from parameter count and storage format, but quantization metadata, embeddings and partitioning can change the loaded footprint. Record what the engine actually reports after startup.

1. Load the model: measure device memory after weights and any adapters are loaded.
2. Reserve runtime headroom: account for execution workspace, graph capture and allocations that appear under load.
3. Compare the remainder with live KV demand: use the busy-hour token mix and the engine’s allocated cache size.
4. Check the pressure case: add longer prompts, more simultaneous requests and a lower-than-expected prefix hit rate.
5. Repeat per worker: tensor, pipeline and data parallelism alter the allocation on each device; do not divide a cluster total evenly without checking the serving topology.

For example, vLLM can infer KV-cache space from a GPU-memory budget or accept an explicit per-GPU cache size. Treat its startup report and out-of-memory behavior as evidence about this configuration, not as a universal capacity figure.

Keep the budget per worker. In data-parallel serving, each replica may load its own weights and reserve its own cache; a cluster total is not one shared pool. In tensor or pipeline parallel serving, memory is distributed differently and some state may be replicated. A vendor quote stating only aggregate HBM across all GPUs can therefore hide a per-worker shortage. Ask for the allocation map for the proposed parallelism and the largest request it can admit at the target concurrency.

Choose a memory tier only after locating the miss

NVIDIA documents host-memory KV offloading as an implementation option. It adds a movement path and can change latency, so test it at the same workload and service target as a GPU-only baseline. An external memory device is not automatically an application-ready KV tier: the engine, placement policy, transfer path and failure behavior must all be specified.

Separate capacity from reuse. If the active decode set must remain close to the GPU, a larger remote cache for old prefixes may reduce repeated prefill work without increasing the number of simultaneous decodes. If the working set spills during active requests, moving blocks back and forth may add pressure to the transfer path. State which behavior the proposal is meant to change and measure that behavior directly.

For the broader architecture choice, see shared AI memory options. That guide compares local, CXL and network-accessed memory without assuming they have the same programming model.

Turn the worksheet into a buying test

Ask every candidate configuration to serve the same model revision and request mix. Fix the offered load and report completed requests, errors, latency and memory use together. Raising throughput by allowing unbounded response times is not an equivalent result.

• Record: model and engine versions, GPU and host configuration, cache data type, parallelism and memory settings.
• Replay: prompt and generation-length distributions, arrival rate, concurrency and shared-prefix frequency.
• Measure: requests per second, output tokens per second, time to first token, time per output token, P95/P99 latency and errors.
• Inspect: GPU and host memory use, cache occupancy, eviction or preemption, prefix hit rate and transfer volume.
• Stress: long-context bursts and a low-reuse case, then document what happens when capacity is exhausted.

vLLM’s serving benchmark reports latency and throughput measures that can be used for this comparison. Publish the inputs with any result so a buyer can distinguish a reproducible test from a headline number.

Run a small test matrix instead of one headline run

A useful comparison has at least three traffic shapes: the expected mix, a long-context burst and a low-reuse case. Repeat each at several offered loads. Hold the model, generated output limits and service target constant while changing one architecture or cache setting at a time. A candidate that looks attractive at light load may fail at the busy-hour arrival rate.

Measure at a common service level. For example, report how many requests complete while satisfying the agreed first-token and per-token latency limits, then show the same measure for each candidate. Also report rejected and timed-out requests. An average latency calculated only from successful requests can conceal a configuration that sheds its hardest traffic.

vLLM documents cache and request metrics that help explain why a run changed, including cache usage and preemption information. Instrumentation varies by engine and version, so record the exact metric names and collection interval in the result. Keep raw logs or dashboard exports alongside the summary rather than presenting a rounded throughput number alone.

Compare cost at the same service target

More memory may let a system carry more concurrent requests, but it also adds hardware, power, software and operations cost. Compare complete configurations at the same latency and availability target. A low price per gigabyte is only useful if the added tier changes the number of successful requests or the hardware needed to deliver them.

• Capacity cost: GPU, host and any additional memory devices, plus network ports and rack power.
• Serving cost: resources reserved for the engine, data movement, monitoring and fault recovery.
• Useful output: requests or output tokens completed within the agreed latency target, including errors and retries in the accounting.
• Operating range: expected utilization and peak headroom, since a system sized only for peak traffic may sit idle much of the day.

Ask the supplier to show its math. If a proposed memory tier claims to avoid buying additional GPUs, identify which GPUs, at what load, and whether the same service target holds in the stress cases. If the comparison needs a future software feature or a particular cache-hit rate, state that dependency explicitly. This makes the decision reviewable without presenting an unverified savings percentage.

Put the worksheet in the procurement brief

The final output is a short, reproducible requirement set rather than a memory number in isolation. Give each vendor the same model and traffic assumptions, then request the complete system boundary: accelerators, host memory, fabric, serving software, cache policy, power and support. If the proposal includes an offload tier, specify who integrates it with the runtime and who responds when it fails.

1. Attach the model configuration and the calculated KV bytes per token, noting any architecture exceptions.
2. State the typical, busy-hour and long-context live-token scenarios, plus admission and latency targets.
3. Request a per-worker memory budget showing loaded weights, runtime reserve and allocated KV cache.
4. Require a test plan with the traffic replay, reported metrics, pass criteria and software versions.
5. Ask for the delivered BOM and responsibility matrix for the GPU, host, network, software and support boundaries.

The GPU server sourcing guide shows how to put system requirements into an RFQ. If the proposed design includes a rack or site change, use the GPU rack power guide to check the electrical handoff as well.

Frequently asked questions

How do I estimate KV-cache memory per token?

For a conventional transformer, multiply 2 × attention layers × KV heads × head dimension × bytes per KV element. The two represents key and value tensors. Then multiply by the tokens held across active requests. For example, 32 layers, 8 KV heads, dimension 128 and 2-byte elements yield 128 KiB per token; 16 requests holding 4,096 tokens each yield 8 GiB of payload. This is a worksheet estimate. The serving engine may round allocation to blocks, reserve workspace or use a different cache layout, so compare it with reported capacity before placing an order.

Should I use the model’s maximum context length for every request?

Use the distribution of live prompt and generated tokens the service will actually hold at the same time. Maximum context length is a useful stress scenario, but applying it to every request may overstate normal demand and conceal the separate question of how to handle unusually long requests. Keep a typical case, busy-hour case and long-context burst case. For the burst, specify whether requests must begin immediately, may be queued or may be routed to a separate pool. That admission policy changes the hardware requirement, so it belongs in the buyer brief.

Does a larger KV cache always improve inference throughput?

A larger cache can admit more simultaneous or longer requests if capacity is the current constraint. It does not make the model compute faster, increase GPU memory bandwidth or fix a poor scheduler setting. Compare completed requests and output tokens per second while holding time-to-first-token and per-token latency targets constant. Also record how many requests are queued, preempted or rejected. If the original configuration has unused cache capacity, adding more memory is unlikely to change the main bottleneck; the next measurement should identify where time is spent.

Can prefix caching reduce the memory I need?

Prefix caching can avoid repeating prefill work when requests share an identical eligible prefix and the relevant blocks remain available. It can also retain blocks after a request finishes, consuming capacity that active requests might need. Do not subtract a guessed percentage from the KV estimate. Measure the real hit rate with representative prompts, then repeat under a busy-hour mix that causes eviction. State whether users or tenants may share cached prefixes, since isolation rules may change what reuse is possible. The result is a cache-policy question as well as a capacity question.

Is CPU or network-attached memory a substitute for GPU HBM?

It can be an additional tier for selected KV blocks or other data, but the serving engine has to know what to move and when. A miss requires data to travel back toward compute, which can affect first-token or tail latency. Ask whether the proposed tier stores reusable completed prefixes, blocks needed during active decode or both; these have different performance demands. Confirm the software connector, transfer bandwidth, placement policy and failure behavior. Compare the full service at the same model, traffic and latency target as a GPU-only baseline.

How much headroom should I leave on each GPU?

There is no safe universal percentage. The loaded weights, adapters, execution workspace, graph capture and cache allocator all consume device memory, and some allocations change under load. Start from the exact engine’s reported memory budget after startup. Run the expected traffic mix, then a busy-hour and long-context burst while watching free memory, cache use, preemptions and errors. Do this per worker: aggregate HBM across a cluster does not show whether one replica has enough space. Keep enough margin to meet the agreed service target under the stress case, not merely to avoid a startup failure.

What metrics should a buyer request from an inference sizing test?

Request the model revision, engine version, hardware, cache format, parallelism, prompts, output lengths, arrival pattern and concurrency so the test can be reproduced. Report offered and completed requests, errors, output tokens per second, time to first token, time per output token and P95/P99 end-to-end latency. Add GPU and host memory, cache occupancy, evictions or preemptions, and observed prefix hits. If the design offloads data, include transfer volume and the added tier’s utilization. Show results at the same service target for each candidate, including any rejected or timed-out requests.

When is another memory tier worth evaluating?

Evaluate another tier when measured live KV demand or reusable prefix data exceeds economical GPU memory and the serving engine supports moving that data. First check whether cache format, prefix reuse, admission limits or a different GPU configuration solves the requirement more simply. Then test the extra tier with the same traffic and latency target as the baseline. State whether the purchase aims to support longer contexts, more concurrent requests or lower cost for the same service level. If the claimed benefit depends on a high cache-hit rate, include a low-reuse test before accepting it.

How do grouped-query attention and quantized KV change the calculation?

Grouped-query attention uses fewer KV heads than query heads, so use the model’s KV-head count in the formula. Multi-query attention can reduce that count further. A lower-byte cache format can reduce the estimated payload, but the actual format must be supported by the model and serving engine; metadata, allocation granularity and quality checks may affect the deployed result. Hybrid attention, sliding windows and multimodal state can require a different calculation altogether. Record the exact model configuration, compute a first estimate, then reconcile it with the engine’s reported cache capacity.

What should an inference-memory supplier quote include?

Ask for a per-worker memory budget, not only the aggregate GPU memory in the cluster. The quote should name the exact model and engine versions, accelerator configuration, host memory, cache data type, parallelism, proposed memory tiers, network requirements and the traffic assumptions used to size them. Require a test plan with pass criteria for throughput, first-token and per-token latency, tail latency and errors. Specify who integrates the serving runtime, who owns cache-policy changes and who diagnoses a failed memory tier. This makes competing designs comparable and exposes dependencies hidden behind a single capacity number.