The VRAM math: weights, a KV cache, and your real context limit

How much VRAM you need to run an LLM comes down to two numbers, not one. The first is fixed the moment you choose a model and a precision — the weights, measured as parameters times bytes per parameter. The second grows with every token you feed it — the key-value (KV) cache. Most guides stop at the first number. The second is the one that decides whether your context window is 8K or 128K.

That gap is why a 70-billion-parameter model that loads in 40 GB can still run out of memory: at 128K tokens of context, its KV cache alone needs roughly 42 GB — as much as the weights. Invert the usual framing and the real question becomes obvious. Once the model fits, the VRAM that’s left over is your context budget. Nothing else.

This guide gives you both formulas, a recomputed sizing table for the model sizes you’ll actually run, and a worked example — a 35B model at 4-bit on a 96 GB card — that turns leftover VRAM into a concrete token count. It also explains why the cloud can advertise a million tokens of context that no single GPU you can buy will hold. The stakes are simple: pick the wrong number and you either overpay for hardware or hit an out-of-memory wall in production. For the wider backdrop on why context lengths exploded in the first place, see our guide to the 10M-token era.

Every VRAM question collapses into the same pair. One number is fixed and one grows. Confusing them is why so much hardware advice is wrong: a card that comfortably loads a model can still be useless for the context length you actually need. Get both into a single spreadsheet row and the rest of this guide is just arithmetic.

The mental model: weights are the cost of admission; the KV cache is the metered tab that runs while you’re inside. You pay the first once and the second per token. Everything that follows — quantization, GPU choice, the gap between local and cloud context lengths — is a consequence of how these two numbers interact inside a fixed VRAM envelope.

The first number is the easy one. Multiply the parameter count by the bytes each parameter occupies at your chosen precision: 4 bytes at FP32, 2 at FP16/BF16, roughly 1 at FP8 or INT8, and about 0.5 at 4-bit. A 70B model is therefore 280 GB at FP32, 140 GB at FP16, 70 GB at INT8, and around 35 GB at 4-bit. Then add 10–20% on top for activation tensors, framework buffers, the CUDA context, and a small KV cache at default context length — a 1.15× multiplier is a safe rule of thumb.

The “Loaded @ 4-bit” column is the number that matters for planning: raw 4-bit weights times 1.15. Note the 70B row — about 40 GB loaded. That is why a 32 GB RTX 5090 cannot hold a 4-bit 70B on a single card no matter how you slice it, while a 96 GB card swallows the weights and still leaves room for a large context. Real 4-bit GGUF formats (Q4_K_M and friends) run slightly above a clean 0.5 bytes per weight, so treat these as floors, not exact reservations. For the quality-versus-size trade behind each precision, see our 4-bit vs 8-bit vs FP8 tradeoff data.

During generation, the model caches the key and value tensors for every token it has already seen so it doesn’t recompute them. That cache is the KV cache, and it grows one token at a time. The total size is governed by a single equation built entirely from the model’s architecture and the context length.

Plug in a Llama 3.1 70B (80 layers, 8 KV heads, 128-dim heads, BF16): 2 × 80 × 8 × 128 × 2 = 327,680 bytes per token, about 0.33 MB. At 128,000 tokens that is ~42 GB; at a million tokens it is ~327 GB. The table below runs the same arithmetic for the models and context lengths you’re most likely to hit, including what happens when you quantize the cache itself.

Two cautions before you build a budget on this table. First, the formula counts a single request; multiply by batch size for concurrent users. Four simultaneous 128K requests on a 70B is 4 × 42 = 168 GB of cache alone, which is why throughput planning is a different problem from single-session context. Second, the cache is also the biggest lever you have: dropping it to an FP8 element halves every cell, and an INT4 cache quarters it. The full toolbox — paged attention, prefix caching, quantized caches — lives in our KV-cache optimization techniques guide.

Here is the calculation almost no guide does in full: take a real card, load a real model, and ask how much context the remaining VRAM actually buys. Work a 35B model at 4-bit on a 96 GB RTX PRO 6000 in three steps and the answer falls out cleanly.

That is the whole insight in one line: a smaller model doesn’t just cost less VRAM, it frees enormous context headroom. The same card that gives a 35B model nearly 290K theoretical tokens gives a 70B far less, because the 70B both weighs more and burns cache faster. Swap the 35B for a 70B at 4-bit (~40 GB loaded) and the headroom drops to about 56 GB — enough for the model’s native 128K window (~42 GB of cache) with a thin margin for activations, and nowhere near the 256K tier. Treat these as theoretical ceilings: a real framework reserves activation and scratch memory too, so usable context lands somewhat below the cache-only math.

Run the budget for a 70B at 4-bit on a 96 GB card and the ceiling is stark. Weights take ~40 GB. A 128K cache adds ~42 GB, for ~82 GB total — it fits, with about 14 GB left for activations and the CUDA context. Push to 256K and the cache alone is ~84 GB; weights plus cache is ~124 GB, over budget. The card tops out near 128K, which also happens to be the model’s trained window. Both limits converge on the same answer.

So how does an API advertise a million tokens? Not by fitting it on one card. That same 70B at 1M needs ~327 GB of KV cache — 3.4× a 96 GB card. Cloud providers reach it by sharding the cache across many GPUs and overlapping the work: context parallelism splits the sequence across devices, ring attention passes key/value blocks around a GPU ring so each device sees the whole context, and cache-offloading plus NVFP4 quantization shrink what has to stay resident. The gap between “what fits on my GPU” and “what the API offers” is architecture and orchestration, not just raw VRAM. We dig into the provider-by-provider numbers in our 1M-to-10M context window comparison.

VRAM capacity tells you whether a model loads. It says nothing about how fast it generates. Inference has two phases with different bottlenecks, and conflating them is the most common buying mistake.

Decode — generating one token at a time — is where you spend most of your runtime, and it is bound by memory bandwidth, not compute. A useful ceiling: theoretical tokens per second is roughly memory bandwidth divided by the bytes the model must read per token, and real-world numbers land at 60–80% of that. Double the bandwidth and you roughly double the speed. Capacity gets the model in the door; bandwidth decides how fast it talks.

The same bandwidth lens explains the rest of the field. A 96 GB RTX PRO 6000 has been independently measured at around 31–32 tok/s on a 70B — the practical reference for a single-card 70B. An Apple M5 Max (614 GB/s, up to 128 GB) lands lower; community benchmarks put a 70B in the ~15–32 tok/s range depending on framework and quantization, so treat any single figure as an estimate. And the fast-but-small RTX 5090, despite matching the PRO 6000’s 1,792 GB/s, can’t hold a 70B at all — point it at models up to ~30B, where its bandwidth implies roughly 60–90 tok/s (an estimate derived from memory bandwidth, not a measured benchmark). The tokens-per-second figures here are approximate and stack-dependent — measured for the PRO 6000, community estimates for the M5 Max, and bandwidth-derived for the 5090.

With the weight math, the cache math, and the bandwidth lens in hand, the hardware choice gets concrete. Match the card to whether your bottleneck is fitting a model, generating fast, or both — and budget for a hardware market that is anything but stable right now.

Quantization is the dial that makes all of this affordable — fewer bytes per parameter shrinks both the weights and, when applied to the cache, the KV footprint. The trade is quality, but on modern formats it is smaller than most people expect until you push past 4-bit.

The quality numbers above are directional — they vary by model, benchmark, and the specific quantization implementation — but the shape is consistent: FP8 and INT8 are close to free, Q4_K_M is the local default that almost nobody can distinguish blind, and only at 2-bit does degradation become obvious. The same logic applies to the cache: an FP8 KV halves every cell in the table above and an INT4 KV quarters it, often with negligible quality cost on long-context retrieval. For the cross-model regression data behind each tier, our quantization tradeoffs breakdown has the benchmarks. And if you’re weighing local inference against a managed deployment for a production workload, that build-versus-buy call is exactly what our AI transformation engagements are built to answer.