Self-Hosting Open-Weight LLMs: When It Actually Wins

Self-hosting open-weight LLMs is the deployment decision teams get wrong most often in 2026 — not because the engineering is hard, but because the cost intuition is backwards. Owning the GPU feels cheaper than paying a per-token API bill, yet for most workloads the math never crosses over, and the costs that sink the project rarely show up on the GPU invoice.

The open-weight ecosystem has never been stronger. Llama 4, the DeepSeek V-series, and the Qwen3 open checkpoints give you frontier- adjacent capability you can run on your own hardware, fine-tune on proprietary data, and operate without an external dependency. The question is no longer whether you can self-host — it is whether you should, and the honest answer depends on volume, utilization, and compliance, not on the model.

This guide gives you the framework: a token-volume breakeven matrix that shows the actual crossover points, the GPU-utilization multiplier that quietly wrecks under-loaded deployments, a 2026 serving-framework decision matrix that reflects the December 2025 sunset of Text Generation Inference, GPU sizing tables by model tier, and the compliance scenarios where the cost math stops mattering entirely. Every figure below is sourced; treat all pricing as a snapshot to re-verify, not a constant.

Most self-hosting debates start in the wrong place. Teams compare benchmark scores, decide an open-weight model is "good enough," and conclude that running it themselves must be cheaper than paying an API per token. The capability question is largely settled — open checkpoints from Llama 4, the DeepSeek V-series, and Qwen3 are genuinely strong. The questions that actually decide the outcome are three: how many tokens you process per month, how busy your GPU stays, and whether regulation forces your hand.

Get those three right and the decision makes itself. A team doing a few hundred million tokens a month at variable load, with no compliance constraint, is almost always cheaper and faster on a managed API. A team processing billions of tokens at steady, high utilization — or one that legally cannot send its data to a third party — is a strong self-hosting candidate. The middle ground is where the matrices below earn their keep.

Three deployment paths exist, and most teams should expect to use more than one. Self-hosting on your own GPUs gives maximum control and the only path to true on-prem data sovereignty. Managed open-weight APIs — Together AI, Fireworks AI, and others — give you open-model weights without the operations burden, at per-token pricing. And local runtimes like Ollama give individual engineers a zero-friction way to prototype before any of this matters.

The single most useful artifact in this decision is a crossover curve: at what monthly token volume does owning the GPU become cheaper than paying per token? The answer depends entirely on which API you are comparing against. Against a premium frontier API, the crossover arrives relatively early. Against a hyper-cheap open-weight API, it may never arrive in practice.

One published analysis frames the extremes cleanly. At roughly 1 million tokens per day, self-hosting a single A100 80GB lands near $3,240 per month all-in — about $1,440 for cloud rental, around $1,500 in DevOps labor, and roughly $300 of infrastructure overhead — versus a budget API bill of only a few dollars at that volume. The crossover against DeepSeek V4 Flash, at about $0.14 per 1M input tokens, does not arrive until roughly 5.7 billion tokens per month. Against a GPT-5-class API near $5 per 1M, a separate estimate puts the crossover closer to 256 million tokens per month.

Read that spread carefully, because it is the whole argument. The cheaper your API alternative, the more tokens you must self-serve before owning hardware pays off — and budget open-weight APIs are now so cheap that the crossover sits in territory most teams will never reach. The reason a budget API can undercut your own H100 is structural: the provider runs that GPU at near-saturation across thousands of customers, so its effective cost-per-token is far below what a single tenant achieves on a partially idle box. The next section is the direct consequence of that fact.

At genuine industrial scale the picture flips hard. The same body of analysis puts self-hosting a Llama 70B at around 500 million tokens per day near $4,360 per month, versus roughly $22,500 per month on managed APIs — about a 5x win for self-hosting. The lesson is not "self-hosting is bad" or "self-hosting is good." It is that the answer is a step function of volume, and you need to find your own row on the curve.

Here is the failure mode that turns a sound-looking self-hosting plan into a money pit. A GPU bills the same whether it is running at 100% or 5% of capacity. Your headline "cost per 1,000 tokens" is only achievable when the card is saturated. The moment your traffic is bursty, overprovisioned, or simply lower than projected, your effective cost-per-token climbs in inverse proportion to load — and it climbs fast.

The arithmetic is unforgiving. If a cloud GPU is running at 10% load, one analysis shows the effective cost per 1,000 tokens jumping from about $0.013 to roughly $0.13 — a 10x increase that makes an under-loaded H100 more expensive per token than a premium API service. The headline rate of about $0.013 per 1K tokens is only valid at high utilization; quote it without the load assumption and you have mispriced the entire project.

This is why managed APIs are so hard to beat at low and medium volume, and it is the mechanism behind the breakeven spread in the previous section. A provider amortizes one physical GPU across many tenants, so the card stays near saturation and its real cost-per- token approaches the theoretical floor. Your single-tenant deployment has to manufacture that same utilization itself, which means real-time autoscaling, request batching, and enough steady traffic to keep the queue full — operational work that does not appear anywhere on the GPU rental invoice.

And utilization is only the most visible hidden cost. The same analysis estimates a 3-5x multiplier over the raw GPU price once you account for DevOps salaries near $145,000 per year, periodic model update cycles around $12,000 each, and ongoing infrastructure overhead. When you model self-hosting, model the whole stack: idle capacity, the engineers who keep it alive, and the upgrade treadmill — not just the line item that is easiest to see.

If your last serving comparison is more than a year old, update it. The biggest change is a quiet one: Hugging Face moved its Text Generation Inference engine into maintenance mode on December 11, 2025, accepting only minor bug fixes and documentation updates and redirecting its engineering effort upstream into vLLM and SGLang. Existing TGI deployments continue to work, but new greenfield production should not start there.

That leaves two clear front-runners and a prototyping tool. vLLM has become the de facto open-source inference standard, with more than 81,000 GitHub stars and over 2,000 contributors as of late May 2026. Its core trick, PagedAttention, manages the KV cache using OS-style virtual-memory paging — and because traditional serving pre-allocates memory for the maximum sequence length and wastes a large share of GPU memory on average requests, vLLM reports up to 24x higher serving throughput than naive static batching. Its continuous (iteration-level) batching keeps the GPU saturated at every forward pass, which the project reports as 3-10x higher throughput on the same hardware.

SGLang is the underreported fork in the road. On standard Llama 3.1 8B workloads on H100 GPUs, one benchmark puts it about 29% ahead of vLLM in raw throughput, with faster time-to-first-token and lower inter-token latency. Its real differentiator is RadixAttention for prefix-heavy workloads: when many requests share a system prompt or a document context — exactly the RAG and multi-turn pattern — it reports throughput gains of up to 6.4x by reusing the shared prefix. For DeepSeek V-series inference specifically, its specialized Multi-head Latent Attention backend is reported around 3.1x faster than vLLM, though under heavy concurrent saturation the two converge.

Do not over-read those numbers into a blanket "SGLang wins." Both engines are excellent and the gap narrows under saturation; vLLM's breadth — 200-plus model architectures and hardware support spanning NVIDIA, AMD ROCm, Google TPU, Intel Gaudi, Apple Silicon, and CPUs — makes it the safer default for a mixed model fleet. The practical heuristic is workload-shaped: vLLM for general and batch throughput and broad hardware coverage, SGLang for prefix-heavy RAG and multi-turn and DeepSeek V-series serving, Ollama for desktop prototyping, and a careful look at NVIDIA's TensorRT-LLM only if you are all-in on NVIDIA and chasing peak FP8 throughput. NVIDIA's own page cites over 10,000 output tokens per second on H100 at FP8 with sub-100ms TTFT, but that is a vendor-stated figure — validate it against independent benchmarks on your model before believing it.

Once you have decided to self-host, the first concrete question is which GPU and how many. The sizing rule is simple to state: multiply the parameter count by the bytes-per-precision, then add 20-40% headroom for activations, the KV cache, and serving overhead. FP16 is 2 bytes per parameter, FP8 is 1 byte, and INT4 is 0.5 bytes. A 70B model at INT4 is roughly 35GB of weights, so about 42GB minimum with 20% overhead — which fits comfortably on a single 80GB card.

Quantization is what makes single-GPU serving of large models tractable. AWQ is the current best-practice INT4 format for vLLM production, identifying activation-salient weights to preserve more quality than GPTQ at the same bit-width, and INT4 methods generally stay within about 6% of baseline perplexity on most tasks. One important caveat: multi-step reasoning and math are more sensitive to quantization quality loss, so validate those use cases explicitly before shipping a 4-bit model into a reasoning-heavy pipeline.

Card choice matters as much as card count, and the decisive spec for large or long-context models is memory, not raw compute. The H100 80GB SXM offers 3.35 TB/s of HBM3 bandwidth; the H200 raises that to 4.8 TB/s of HBM3e — about 43% more bandwidth — and lifts capacity to 141GB, roughly 76% more VRAM. That extra capacity is the H200's real advantage for 100B-plus models and long-context serving, where the KV cache alone can consume tens of gigabytes. By contrast, the L40S uses 48GB of GDDR6 at around 846 GB/s — roughly 4x less bandwidth than an H100 — and is positioned for cost-sensitive inference of smaller models where GDDR6's lower cost per GB offsets the bandwidth gap.

Keep the KV cache in view when you size. A 70B model at a 128K-token context can need on the order of 42GB of KV cache by itself, on top of the weights — which is exactly why long-context workloads push you toward higher-VRAM cards or more aggressive compression. And the GPU need not be NVIDIA at all: AMD ROCm is now a first-class vLLM target, with roughly 93% of the vLLM AMD test suite passing as of January 2026, so teams are no longer locked to a single vendor for production inference. If you are weighing the full picture against managed options, our AI digital transformation engagements start with exactly this kind of comparative deployment eval.

For one class of organization the entire breakeven discussion is beside the point. If you process regulated data — protected health information, EU personal data at scale, or anything under a strict SOC2 boundary — the question is not whether self-hosting is cheaper. It is whether you are legally permitted to send that data to a third-party API at all. Often the answer is no, and that makes a self-hosted or VPC deployment the only straightforward compliant path, regardless of token volume.

HIPAA is the clearest example. Processing protected health information through any provider acting as a business associate requires a Business Associate Agreement, and standard consumer APIs from the major frontier vendors lack BAA coverage by default. Transmitting PHI to such a service without a BAA in place is itself a compliance violation — which is why clinical and billing workloads so often land on self-hosted infrastructure even when the per-token math would favor an API.

The practical takeaway is to sequence the questions correctly. Ask the compliance question before the cost question. If a BAA, an SCC gap, or a data-residency requirement removes managed APIs from the table, you self-host — and the matrices above stop being about "whether" and start being about "which GPU and which framework." If compliance is not a constraint, then the cost crossover and utilization math run the decision.

Putting it together yields a short, opinionated decision tree. Most teams will land in one of four buckets, and the right move differs sharply between them. The mistake to avoid is treating self-hosting as a status decision rather than an economic and regulatory one.

Whichever bucket you fall into, the discipline is the same: run a real A/B before committing capital. Stand up the open-weight model on a managed API, measure your true monthly token volume and your achievable GPU utilization, and only then price a self-hosted deployment against that baseline using live cloud GPU rates. The numbers in this guide are a map of the terrain — your own measurements are the route. For a deeper look at the per-token economics behind the breakeven curves, our current token pricing across frontier and open-weight models and the broader open-weight versus closed-source model tradeoffs give the surrounding context.

If your self-hosting case is driven by RAG, the supporting infrastructure decisions matter as much as the serving engine — choosing the right vector database for RAG applications and modeling AI inference cost optimization end to end will move your effective cost more than a 29% framework throughput edge ever will.