DeepSeek V4 Launches: 1.6T MoE, 1M Context, 10% KV

DeepSeek published the DeepSeek-V4 Preview on April 24, 2026 — a new open-weight Mixture-of-Experts series that stakes the lab's thesis on a single claim: million-token context processing is no longer a capability problem, it's an efficiency problem.

Two models shipped today. V4-Pro packs 1.6 trillion total parameters with 49 billion activated per token. V4-Flash is the efficient sibling at 284 billion total and 13 billion active. Both support native 1M context, both are open weights on Hugging Face, and both are already live on the DeepSeek API and chat.deepseek.com as Expert Mode and Instant Mode respectively.

This guide covers what actually launched, the attention architecture that makes the efficiency story real, honest benchmark positioning versus GPT-5.4, Gemini-3.1-Pro, and Claude Opus 4.6, and how to start using V4 in your own stack today. Everything below is sourced from DeepSeek's technical report published alongside the launch.

The launch is a Preview release — DeepSeek's term for a production model that the lab considers complete enough for API traffic and open distribution, but that may still evolve before a full "V4" branding. Three surfaces went live simultaneously: open weights on Hugging Face, the DeepSeek API updated to the new models, and chat.deepseek.com exposing V4-Pro as Expert Mode and V4-Flash as Instant Mode.

Both models share a single architectural stack. The differences are the expert count, the hidden size, and the training budget. The paper is explicit that Flash-Base already surpasses V3.2-Base across the majority of benchmarks despite roughly 42% of V3.2's total parameter count — largely because of the architectural and data-quality improvements documented in the sections below.

The release follows the pattern DeepSeek established with V3.1 and V3.2: permissively-licensed open weights, a detailed technical report published alongside the release, and a reference inference implementation in the same repository. Always verify the exact license text on the Hugging Face repo before shipping production workloads.

The central claim of the V4 paper sits in a single sentence of the abstract: at a one-million-token context length, V4-Pro requires only 27% of the single-token inference FLOPs and 10% of the KV cache compared with V3.2. V4-Flash goes further — 10% of the FLOPs and 7% of the KV cache. Those are not theoretical numbers; they are DeepSeek's measurements of equivalent FP8 FLOPs against its own prior-generation model on the same hardware.

The practical implication is worth reading twice. V4-Pro has more than double the active parameters of V3.2 (49B vs 37B) and more than twice the total parameters. By every intuition built on dense transformer scaling, inference should cost more — not ~3.7× less. The efficiency gain comes entirely from the attention redesign in Section 03, combined with FP4-quantized routed experts and a fundamentally different KV cache management strategy.

For teams operating long-context workloads — full-codebase analysis, multi-document reasoning, legal or financial corpus Q&A — this changes the math. A workload that required $1 of compute on V3.2 at 1M tokens costs roughly $0.27 on V4-Pro with meaningfully stronger capability, or roughly $0.10 on V4-Flash with modestly weaker knowledge but comparable reasoning. That is the entire thesis of this release.

DeepSeek-V4's attention stack interleaves two distinct compression strategies across layers. Neither is exotic in isolation; the contribution is the hybrid configuration and the engineering to make it train stably at this scale.

Compressed Sparse Attention (CSA)

What it is. CSA takes the Key-Value cache for every m tokens and compresses that block into a single entry — so the effective KV sequence is 1/m the length of the raw token sequence. Then, for each query token, a learned Lightning Indexer scores those compressed blocks and a top-k selector picks only the most relevant ones to attend to. A sliding window of recent uncompressed tokens is concatenated alongside so that local fine-grained dependencies are preserved.

Why it matters. Two savings stack: the KV cache itself is 1/m the size of a dense attention KV, and even among the compressed entries, attention is sparse (top-k only). CSA is a generalization of DeepSeek Sparse Attention from V3.2 — same idea, more aggressive compression.

Heavily Compressed Attention (HCA)

What it is. HCA applies much more aggressive compression — every m' tokens (with m' ≫ m) fold into a single KV entry. The trade-off is that HCA skips the sparse selection step entirely; attention over the compressed KV remains dense.

Why it matters. HCA is designed for the layers where retaining a broad, low-resolution view of the full context is more valuable than fine-grained selection. Interleaving HCA blocks with CSA blocks gives the model both modes — precise look-up on some layers, smeared global summary on others.

Beyond the attention redesign, V4 introduces two further innovations and retains the best parts of the V3 stack. The net effect is a model that trains more stably at larger scale than any prior DeepSeek release.

Manifold-Constrained Hyper-Connections (mHC)

mHC replaces the conventional residual connections between Transformer blocks. In standard Hyper-Connections, the residual mapping can amplify signals in unstable ways when many layers stack — causing numerical blow-ups during training. mHC constrains the residual mapping to lie on the Birkhoff polytope (the manifold of doubly stochastic matrices), which bounds the spectral norm to ≤ 1 and makes signal propagation non-expansive by construction. The result: signals stay numerically stable across very deep stacks, which is what unlocks the 1.6T parameter scale at all.

Muon Optimizer

V4 trains with Muonrather than AdamW. In DeepSeek's setup, Muon delivers faster convergence and better training stability, though the paper is careful to note that several training-time stabilizers — Anticipatory Routing and SwiGLU clamping — were still required to keep loss spikes under control at scale.

Carried over from V3

• DeepSeekMoE — the fine-grained routed-expert FFN framework, with a small tweak: V4 uses Sqrt(Softplus) rather than Sigmoid for affinity scoring, removes the cap on routing target nodes, and replaces the dense FFN layers in the first few Transformer blocks with Hash-routed MoE layers.
• Multi-Token Prediction (MTP) — retained unchanged from V3. Still used to accelerate inference and to improve training signal.
• Auxiliary-loss-free load balancing — with a mild sequence-wise balance loss added on top to avoid extreme expert imbalance inside individual sequences.

V4-Pro is pre-trained on 33 trillion tokens, V4-Flash on 32 trillion. Both training runs use FP4 quantization-aware training for the routed expert weights and the indexer query/key path, while keeping non-expert computation in FP8. The practical effect today is a smaller memory footprint during training and inference; the paper notes that on current hardware, peak throughput for FP4×FP8 operations is identical to FP8×FP8, but explicitly flags that purpose-built hardware could make FP4 roughly 1.33× more efficient than FP8 — an open lane for future inference gains.

Training stability was actively managed. The paper introduces Anticipatory Routing, which decouples routing updates from the backbone network by one step, fetched in advance — triggered automatically when a loss spike is detected. Combined with SwiGLU clamping (the linear component clamped to [-10, 10], upper gate capped at 10), the authors were able to avoid loss-spike recovery without compromising final-model quality.

V4's post-training pipeline is the second big break from V3-series practice. The mixed-RL stage — previously used to consolidate capabilities across domains — is entirely replaced by On-Policy Distillation (OPD).

The sequence: train a separate specialist model for each target domain (math, code, agent, instruction following) via Supervised Fine-Tuning followed by Reinforcement Learning using Group Relative Policy Optimization (GRPO). Those specialists each become state-of-the-art in their respective field. Then train a single unified model via multi-teacher OPD, where the unified model is the student and the specialists are teachers — the student optimizes a reverse-KL loss against teacher output distributions on its own generated trajectories. The result: one model that inherits the specialists' capabilities without their per-domain narrowness.

Three reasoning modes

Each V4 model supports three inference modes, distinguished by how they use the <think> / </think> tokens:

Think Max is what produces V4-Pro-Max's strongest benchmark numbers in Section 07. At inference time, Max mode adds a prepended instruction demanding "absolute maximum" reasoning — explicitly decomposing the problem, documenting rejected hypotheses, stress-testing against edge cases — and uses a meaningfully larger context budget and reduced length penalty than High mode. The cost is output token count; the payoff is the frontier-competitive numbers on hard reasoning.

Two operational changes worth knowing

• DSML XML tool-call schema.V4 replaces V3.2's tool-call format with an XML-based schema using dedicated |DSML| tokens. The paper reports fewer escaping failures and tool-call errors; practically, any agent scaffolding calling V4 should expect a slightly different tool-invocation format than V3.2 used.
• Interleaved Thinking. Unlike V3.2, which discarded reasoning traces at the start of each new user turn, V4 retains the complete reasoning history across tool calls and user messages during tool-using conversations. That preserves coherent long-horizon chains of thought for agent tasks, at the cost of more context consumption.

The chart below is a direct subset of Table 6 from the V4 paper, comparing V4-Pro-Max against the strongest publicly evaluated modes of Claude Opus 4.6 (Max), GPT-5.4 (xHigh), and Gemini-3.1-Pro (High). Orange bars mark V4-Pro-Max scores where V4 leads the field; blue bars mark where a closed-frontier model leads.

DeepSeek's own framing is the honest version: V4 trails the absolute frontier by approximately 3 to 6 months on general knowledge and the hardest retrieval workloads, while setting new open-model highs on competitive programming and formal reasoning. Open Codeforces rating of 3206 places the model roughly 23rd among human contest participants. The 120/120 Putnam-2025 is proof-perfect — every solution a valid, graded proof rather than a numeric answer.

Three paths are live today. Pick the one that matches the workload: open weights for on-prem or fine-tuning, the API for production integration, or the chat UI for exploration and team evaluation.

For organizations with data-sovereignty or sector-compliance requirements, V4's open weights plus the 1M-context efficiency story make it the strongest open candidate today for on-prem long-document RAG replacement. For most agencies and engineering teams, the practical starting point is the API and chat surfaces — benchmark on your own prompts, measure token spend and latency, decide per-workload before considering on-prem. If you're deciding between V4 and closed frontier for specific pipelines, our AI digital transformation engagements start with exactly this kind of comparative eval.

V4's release changes the practical decision tree for three specific workload classes. For everything else, the frontier picture hasn't moved.

For teams currently running V3.2 in production, the migration is straightforward — same architectural family, same weight-distribution story, meaningfully lower inference cost at 1M context, and three reasoning modes that replace the per-task prompt engineering V3.2 typically needed. For teams evaluating open weights for the first time because of sovereignty or cost, V4-Flash is the right starting point: 284B total / 13B active is tractable on modest clusters, and Flash-Base already surpasses V3.2-Base on most benchmarks.