Size in tokens, not TFLOPS
The useful line item is HBM remaining after weights, divided by bytes per token — a figure a facility can commit to and a customer can be sold.
INSIGHTInference economics
The KV cache is why inference is memory-bound
In large-language-model serving the binding constraint is usually memory, not arithmetic. The key-value (KV) cache grows linearly with context length and with the number of concurrent sequences, and — unlike model weights — it is not shared across a batch. On an eight-GPU H200 node serving a Llama-3-70B-shaped model in FP16, the cache overtakes the model itself at roughly 427,000 tokens of live context, and the attention step that reads it runs at about 4% of the accelerator's FLOP-per-byte ridge point. Capacity planning that counts TFLOPS will size the wrong machine.
PUBLISHED LAST VERIFIED BY JOSEF ELIMELECHREVIEWED PODOS AI ENGINEERING
- 320 KiB
- KV per token · 70B shape, FP16
- 427,000
- Tokens before cache exceeds weights
- 8 FLOP/B
- Decode-attention intensity
What you need to know
01
Cache size is arithmetic
2 x layers x kv_heads x head_dim x bytes, times context, times concurrency. No benchmark required — published model shapes give an exact answer.
02
It does not amortise
Batching reads the weights once for the whole batch. Every sequence still reads its own cache, in full, on every step.
03
Concurrency is the ceiling
HBM left after the weights, divided by bytes per token, is the number of sequences a node can hold. Clock speed does not enter into it.
04
The fix is bytes, not FLOPs
Grouped-query attention, latent attention, and KV quantization all buy the same thing: fewer bytes per cached token.
The short answer
The KV cache binds LLM serving because it is per-sequence and read in full on every decode step, so batching amortises the weights but multiplies the cache. Its size is arithmetic, not a benchmark: 2 × layers × kv_heads × head_dim × bytes, times context, times concurrency. On the published Llama-3-70B shape — 80 layers, 8 KV heads, head dim 128[5] — that is 320 KiB per token in FP16, so an eight-GPU H200 node with 1,128 GB of HBM3e[6]has roughly 815 GiB left after weights and overhead: about 20 concurrent 128K-token sequences, and a cache that outweighs the model itself past ~427,000 tokens of live context. Meanwhile the decode-attention step reads those bytes at an arithmetic intensity of exactly q_heads/kv_heads — 8 FLOP per byte against a ~206 FLOP/byte ridge point, or 3.9% of the machine's arithmetic capability. A capacity model denominated in TFLOPS cannot see any of this, and will size the wrong machine.
Plain English first
What the cache actually is, and why its size is not negotiable
A transformer generates one token at a time, and every new token attends to every token before it. Rather than recompute the representation of the whole conversation on each step, the server keeps two vectors per token, per layer — a key and a value — and re-reads them. That store is the KV cache. It is not optional, and its size is not a matter of tuning; it is arithmetic:
bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_element
cache bytes = bytes_per_token × context_length × concurrent_sequences
Nothing in that expression is a benchmark. Put a published model shape into it and the answer is exact. Meta publishes the shapes for Llama 3: 32 / 80 / 126 layers at 8B / 70B / 405B, model dimensions of 4,096 / 8,192 / 16,384, 32 / 64 / 128 attention heads, and — critically — 8 key/value heads at every size.[5] Head dimension is model dimension over attention heads, which is 128 in all three cases.
Fig. 1 · Derived from [5]
What one cached token costs
Head dim 128, two bytes per element. The last column is the counterfactual if the same shape used one KV head per query head — the reason grouped-query attention exists at all.
| Shape | Layers | Q heads | KV heads | FP16 KV per token | Same shape, full MHA |
|---|---|---|---|---|---|
| 8B | 32 | 32 | 8 | 128 KiB | 512 KiB |
| 70B | 80 | 64 | 8 | 320 KiB | 2,560 KiB |
| 405B | 126 | 128 | 8 | 504 KiB | 8,064 KiB |
The worked calculation
Assumptions on the surface, not buried in a spreadsheet
Shazeer named the underlying problem in 2019: incremental inference is often slow "due to the memory-bandwidth cost of repeatedly loading the large keys and values tensors," and sharing one KV head across all query heads shrinks those tensors directly.[1] GQA generalised that to an intermediate number of KV heads, reaching quality close to full multi-head attention at multi-query speed.[2] Pope et al. quantified the payoff in the currency that matters here: the lower memory requirement of multiquery attention enabled scaling up to 32× larger context lengths.[3]
So take a 70B-shaped model on a node of eight H200 SXM GPUs. NVIDIA lists 141GB of HBM3e and 4.8TB/s per GPU.[6] The assumptions below are the ones worth arguing with.
- Model weights held in FP16: 70B × 2 bytes = 140 GB (130 GiB).
- Node HBM: 8 × 141 GB = 1,128 GB (1,050 GiB).
- Reserve 10% of node HBM (105 GiB) for activations, workspace, graph buffers and allocator overhead.
- Remaining KV budget: 1,050 − 130 − 105 ≈ 815 GiB.
- KV in FP16 — no quantization, no prefix sharing, no offload to host memory.
- Cache sharded across the eight GPUs by tensor parallelism, so the pool is the node, not one GPU.
Fig. 2 · KV cache, 70B shape, FP16
Cache footprint by context length and concurrency
Emphasised cells exceed the 815 GiB node budget derived above. They do not fit, at any clock speed.
| Context length | 1 sequence | 8 sequences | 32 sequences | 128 sequences |
|---|---|---|---|---|
| 4,096 (4K) | 1.25 GiB | 10 GiB | 40 GiB | 160 GiB |
| 32,768 (32K) | 10 GiB | 80 GiB | 320 GiB | 1,280 GiB |
| 131,072 (128K) | 40 GiB | 320 GiB | 1,280 GiB | 5,120 GiB |
Reading the table
Three numbers a FLOPs-based capacity model cannot see
First, the concurrency ceiling: 815 GiB divided by 40 GiB per full-context sequence is about 20 concurrent 128K-token sequences on that node — 81 at 32K, roughly 650 at 4K. Second, the crossover point: at 320 KiB per token, the cache equals the 130 GiB of FP16 weights at about 427,000 tokens of live context. That is only 3.3 sequences at 128K, 13 at 32K, or 104 at 4K. Past that line the cache — not the model — is the larger tenant of the node's HBM. Third, the shape of the growth: the cache is linear in context and linear in concurrency, so it is quadratic in "serve twice as many users at twice the context."
These are ceilings, not achievable operating points. Real allocators do worse: the vLLM authors measured that "only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems" before paged allocation, the rest lost to internal and external fragmentation.[4] Paged management recovers most of that headroom, but it recovers it against the same hard ceiling — it does not raise it.
The arithmetic intensity of decode attention is simply the grouped-query ratio. Head dimension cancels. Context length cancels. Layer count cancels. Batch size cancels.
8
FLOP per byte · Llama 3 shape
The derivation
Batching rescues the weights and does nothing for the cache
The standard answer to a memory-bandwidth problem is to batch: read the weights once and amortise them across many sequences. That works for the weights. It does nothing for the cache — every sequence has its own, and every sequence reads all of it, every step. Batching multiplies KV traffic instead of amortising it. Per layer, per sequence, per generated token, over a context of length L:
bytes read = 2 × kv_heads × head_dim × L × 2 B = 4 · kv_heads · head_dim · L
FLOPs (QKᵀ then AV) = 2 × (2 · q_heads · head_dim · L) = 4 · q_heads · head_dim · L
intensity = FLOPs / bytes = q_heads / kv_heads
Compare that to the ridge point of the hardware. NVIDIA lists 1,979 FP16 tensor-core TFLOPS for H200 with sparsity — roughly 990 dense — against 4.8TB/s, giving about 206 FLOP per byte before the tensor cores are saturated.[6]
Fig. 3 · Derived, not measured
Every attention scheme is memory-bound; they differ only in how badly
A 64-query-head shape against the 206 FLOP/byte ridge point. Group sizes from [1][2][5]; ridge point from [6].
| Scheme | KV heads | FLOP per byte | Share of the ridge point |
|---|---|---|---|
| MHA | 64 | 1 | 0.5% |
| GQA 8:1 (Llama 3 shape) | 8 | 8 | 3.9% |
| MQA | 1 | 64 | 31% |
Grouped-query attention moves a 70B-shaped model eight times closer to the ridge and still leaves it running attention at about 4% of the machine's arithmetic capability. This is the real content of the claim that inference is memory-bound: not a benchmark result, but a structural property of the operation. Which is also why the interesting architectural work attacks bytes rather than operations — DeepSeek-V2's multi-head latent attention reports a 93.3% reduction in KV cache by compressing keys and values into a latent vector,[9] and TurboQuant reports quality-neutral KV quantization at about 3.5 bits per channel, with marginal degradation at 2.5.[10] Both buy the same thing: bytes.
Consequences
What this means for operators
If the constraint is memory, then the capacity model, the purchasing decision, and the telemetry all move.
Context is a capacity purchase
Moving a product from 32K to 128K does not cost 4× the compute. On the numbers above it costs 4× the cache and cuts concurrency per node by the same factor.
Provision the memory domain
Tensor-parallel serving splits the cache across accelerators sharing one high-bandwidth GPU-to-GPU fabric, so the ceiling is set by the NVLink domain — a rack-scale quantity in designs where 72 GPUs act as one domain.[8]
Do not de-rate power or cooling
Prefill is compute-bound and interleaves with decode on the same silicon. The thermal and electrical design still has to carry nameplate draw.
Instrument HBM occupancy
Cache pressure, not GPU utilization, is what precedes queueing and eviction in a serving fleet — and a utilization dashboard cannot see it.
Re-run the arithmetic per model
Two models of the same parameter count can differ eightfold in bytes per token. The KV shape, not the parameter count, is what a capacity model needs.
In the product
Why this shapes the unit, not just the server
The infrastructure consequence is unglamorous: the memory you can power and cool inside one coherent GPU-to-GPU domain[7][8] sets the inference capacity of a site. That is the design problem behind high-density GPU infrastructure — packing accelerators tightly enough to share one fabric, then removing the heat that packing creates through direct-to-chip liquid cooling and feeding it with a power architecture sized for the worst-case phase.
Each PODOS Pod is designed as a standardized 1 MW building block and designed for 128 GPUs — a unit sized so that power, cooling and the accelerator domain scale together rather than being renegotiated per site. To sketch capacity against your own workload, start with the configurator; unfamiliar terms are defined in the AI infrastructure glossary.
Honest limits
What this does not prove
The calculations above are arithmetic on published specifications. No hardware was measured for this article, and the following limits are load-bearing.
- It is not a benchmark. Figures 1–3 are derived from vendor spec sheets and a published hyperparameter table, not from a serving run. Treat them as ceilings and ratios, never as throughput or latency predictions.
- The dense FLOPS figure is inferred. NVIDIA publishes 1,979 FP16 TFLOPS with sparsity; the ~990 dense figure follows the usual 2× convention rather than a separately published number, and no real kernel reaches peak. A lower dense figure moves the ridge point down, making attention look less memory-bound, not more.
- The FP16 assumption dominates the result. KV quantization to about 3.5 bits per channel would cut the cache roughly 4.5× and raise decode intensity by the same factor; anyone quoting these tables at a different precision is quoting the wrong tables.
- Linear-in-context is an architectural assumption, not a law. Latent attention, sliding-window and hybrid attention, and state-space layers all break the linear model — for those, the crossover arithmetic must be redone from the actual per-token byte count.
- The node budget is an idealisation. It ignores fragmentation, which pre-paging systems suffered heavily, and ignores prefix sharing and offload, which cut the other way. Real usable headroom sits below 815 GiB and depends on the serving stack.
- It says nothing about which regime dominates a given workload. Prefill is compute-bound, and nothing here establishes the prefill/decode mix for any particular deployment.
- For the conclusion to fail, one of two things must become true: accelerator memory capacity and bandwidth would have to grow faster than context windows and concurrency, or attention would have to stop re-reading a per-sequence cache. Neither has happened yet; both are worth watching.
Sources
- [1] Fast Transformer Decoding: One Write-Head is All You Need (arXiv:1911.02150) — Noam Shazeer, Nov 2019
- [2] GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arXiv:2305.13245) — Ainslie et al., May 2023
- [3] Efficiently Scaling Transformer Inference (arXiv:2211.05102) — Pope et al., Google, Nov 2022
- [4] Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv:2309.06180, SOSP 2023) — Kwon et al., Sep 2023
- [5] The Llama 3 Herd of Models — Table 3, key hyperparameters (arXiv:2407.21783) — Llama Team, Meta AI, Jul 2024
- [6] H200 Tensor Core GPU product page — 141GB HBM3e, 4.8TB/s, 1,979 FP16 TFLOPS (with sparsity) — NVIDIA, accessed 2026-08-31
- [7] NVLink and NVLink Switch product page — NVIDIA, accessed 2026-08-31
- [8] GB200 NVL72 product page — NVIDIA, accessed 2026-08-31
- [9] DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv:2405.04434) — DeepSeek-AI, May 2024
- [10] TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (arXiv:2504.19874) — Zandieh et al., Apr 2025
Bring your own token budget
Send the model shape, the context window, and the concurrency you need to serve. Engineering will work the memory arithmetic back to a unit count.