MoE Architecture in 2026: GPT, Claude, DeepSeek, Qwen Compared

By April 2026, mixture-of-experts is no longer the contrarian architectural bet. Every frontier model that ships open weights, and every model rumored to power closed-API frontier (with the explicit exception of the Claude family), is dense-MoE — a sparse ensemble where each forward pass routes through a small subset of total parameters.

The shift happened fast. In 2024, Mixtral 8x22B was the open-weight standard at 28% sparsity. In 2025, DeepSeek V3 dropped that to 5.4%. By Q2 2026, DeepSeek V4-Pro pushes it to 3.1% (49B active out of 1.6T total) and Qwen 3's 235B-MoE variant lands at 9.4%. The consequence is that frontier models now hold ten to thirty times more parameters in VRAM than a dense model of equivalent throughput — and the question is no longer whether MoE wins, but which MoE pattern wins for which serving constraint.

This post compares the four canonical 2026 MoE patterns side by side: the rumored GPT-5.5 architecture, DeepSeek V4-Pro's published spec, Qwen 3's public model cards, and the open-weight reference points (Mixtral, Llama-MoE, Granite-MoE). It also covers Anthropic's notable hold-out — Claude Opus 4.7 is the only frontier model where the public evidence still suggests a dense (or very lightly-MoE) architecture.

The MoE thesis is mechanical. A dense model with N parameters performs N FLOPs per token at inference. An MoE model with N total parameters and k% sparsity performs roughly k% × N FLOPs per token — because only the activated experts contribute to the forward pass. Holding N constant, a 5% sparse model is twenty times cheaper per token to compute than its dense equivalent.

That trade was a research curiosity until two things converged: knowledge density stopped scaling well past about 70B dense parameters (the Llama 3 70B / 405B gap is the canonical evidence), and serving infrastructure matured to the point that all-to-all expert routing across 8 or 16 GPUs became practical at production tail-latency. The combination meant frontier teams could keep adding knowledge in the form of more experts without paying for it on every token.

The trend line is straight: roughly a 9× compression in sparsity ratio over two years. Every step preserved or improved downstream evaluation while reducing per-token FLOPs. The architectural moves that made it work — fine-grained expert split, shared experts for general knowledge, latent-attention compression for KV cache — are now standard, and the next frontier of compression is in the attention layer (MLA, CSA/HCA) rather than the FFN layer where the experts live.

The four side-by-side specs below are what we work from when sizing client deployments. Numbers are taken from official model cards (DeepSeek V4, Qwen 3) or from the most credible public reverse-eng available as of Apr 24, 2026 (GPT-5.5 architecture is not officially disclosed; the figures listed are widely-cited estimates).

Sparsity is one variable; routing is the other. Three families dominate 2026 production: top-k token-choice, expert-choice (the Switch family), and fine-grained shared-experts (the DeepSeek/Qwen pattern). Each family has a characteristic failure mode under bursty traffic, and that failure mode is what governs which stack you can serve cheaply.

Sparsity ratio sets the per-token compute floor. The math: a 1.6T model at 3.1% sparsity does roughly the FLOPs of a 50B dense model on each token. A 235B model at 9.4% sparsity does roughly the FLOPs of a 22B dense model. That ratio shows up directly in tokens-per-second on identical hardware — and indirectly in $/1M tokens through provider markup.

The trap is treating sparsity ratio as a pure win. Lower sparsity buys cheaper per-token compute, but the trade-off shows up in three places: VRAM resident (the full 1.6T must fit somewhere), inter-GPU traffic (expert all-to-all on 8 GPUs is bandwidth-heavy), and tail-latency variance (more experts means more potential for imbalance). Each of these costs money in production — not all of it on the per-token line.

From a serving perspective, MoE is not a strict win — it is a trade. The compute savings are real, but they shift the hot path from FLOPs to two other resources: VRAM (because all experts must be resident) and inter-GPU bandwidth (because expert routing is inherently all-to-all). A serving stack that does not handle these two correctly will throw away most of the per-token compute savings.

• VRAM resident. DeepSeek V4-Pro at FP8 needs ~1.6 TB of VRAM just for parameters, plus another 200 GB for KV cache and activations at long context. That maps to 8×H100 (640 GB at FP8 with weight-only quant) or 8×H200 (768 GB) as the minimum cluster size for FP8 serving.
• Inter-GPU bandwidth. Every token triggers expert all-to-all communication. At 49B active across 256 experts spread over 8 GPUs, this pushes 30-40 GB/s per token of cross-GPU traffic. NVLink-NVSwitch handles it; cheap PCIe clusters do not — and that is the dominant reason frontier MoE is hard to self-host.
• Capacity tuning. Each expert has a configurable capacity (max tokens it will accept per batch). Set too low, tokens get dropped or routed to a fall-back path. Set too high, memory wastes. vLLM 0.7+ exposes this as expert_capacity; SGLang as experts_per_token; production stacks tune this per-workload.

For most agency or product teams the choice collapses to four options: closed-API frontier (GPT-5.5 or Claude Opus 4.7), open-weight frontier MoE (DeepSeek V4-Pro), open-weight mid-cluster MoE (Qwen 3 235B-MoE or Llama 4-MoE 70B), or single-box MoE (Mixtral 8x22B). The decision is governed by token volume, latency target, and how much inference engineering the team can sustain.

The mistake we see most often is a Profile-A team — under 600M tokens/month, no infra engineer — spending three weeks evaluating DeepSeek V4-Pro self-hosting because the per-token rack rate looks irresistible. At their volume, the 8×H100 cluster cost would dominate spend by a factor of four, before counting engineer time. The right answer for that team is the closed-API rack rate plus aggressive caching — not heroic self-hosting.

The opposite mistake is Profile C — 5B+ tokens/month with a dedicated infra team — running everything on closed APIs because the migration feels expensive. At that volume, the migration pays for itself in the first month, every month. We have helped clients cut $1.8M/year in inference spend with a six-week move from closed-API to V4-Pro on 16×H100.