NVIDIA Dynamo 1.0: Open-Source Inference OS for AI

What Is NVIDIA Dynamo

NVIDIA Dynamo is a distributed inference runtime designed to coordinate LLM serving across multi-node GPU clusters. The "inference operating system" framing is deliberate: just as an operating system abstracts hardware resources and schedules processes, Dynamo abstracts a pool of GPUs and schedules inference workloads across them with the goal of maximizing throughput while meeting latency targets.

Traditional inference servers are designed to run on a single machine. As models grow larger and require multiple GPUs, these servers scale up by adding more GPUs to one node and using tensor parallelism to split the model across them. Dynamo takes a different approach: it decomposes the inference pipeline into functional stages — prefill, decode, KV cache management, and routing — and distributes each stage across dedicated worker pools that can span many nodes. This disaggregated architecture is the foundational innovation that enables the system's performance characteristics.

The timing of Dynamo's release reflects a structural shift in the AI industry. Through 2023 and 2024, most AI deployments were relatively small-scale: a few GPUs serving one or two models for internal tooling or early products. In 2025 and 2026, the landscape has changed. Enterprises are operating what NVIDIA calls AI factories — large-scale GPU clusters continuously running inference workloads for customer-facing products, internal automation, and AI agents operating at high throughput. The infrastructure energy demands of AI at scale make efficiency improvements in inference compute directly translate to lower operating costs and reduced power consumption.

Disaggregated Prefill-Decode Architecture

The core architectural innovation in Dynamo is disaggregating the two phases of LLM inference — prefill and decode — onto separate GPU pools. Understanding why this matters requires understanding the fundamentally different computational demands of each phase.

Prefill processes the entire input prompt in parallel. For a 1,000 token prompt, all 1,000 tokens are fed through the model simultaneously in large matrix multiplications. This is highly compute-bound: the bottleneck is raw arithmetic throughput (FLOPS), and GPUs can achieve high utilization by batching many prefill requests together. Decode generates one output token at a time, requiring a complete forward pass through the model for every token produced. With most of the model's weights already loaded into memory but only one token being processed, this phase is memory-bandwidth-bound — the bottleneck is how quickly the GPU can read model weights from VRAM, not how many arithmetic operations it can perform per second.

When both phases share the same GPU, each phase degrades the other's efficiency. Prefill batches are interrupted by decode requests that arrive mid-computation. Decode workers sit partially idle during high-prefill-load periods. GPU hardware optimized for one phase's bottleneck is suboptimal for the other's. Disaggregation allows each worker pool to be configured, scaled, and batched independently — using compute-dense GPU configurations for prefill workers and memory-bandwidth-optimized configurations for decode workers, each running at near-peak efficiency for their specific computational profile.

Distributed KV Cache and Smart Routing

The KV cache — short for key-value cache — stores the attention mechanism's intermediate computation results from the prefill phase so that the decode phase can access them without recomputation. In single-node inference, this cache lives in the GPU's local VRAM and is scoped to a single request. When a request finishes, its cache entries are evicted. When a similar request arrives later with the same prefix, the computation is repeated from scratch.

Dynamo introduces a distributed KV cache layer that is shared across all worker nodes in the cluster. When a request is processed, its KV states for any repeated prefix — such as a common system prompt, a frequently retrieved RAG context chunk, or a shared conversation history — are stored in the distributed cache. When future requests arrive with the same prefix, the cached states are retrieved and injected directly into the decode workers, completely skipping the prefill computation for the matching portion of the input.

The smart routing layer integrates with the KV cache manager to make routing decisions that optimize for cache locality, worker load balance, and request latency simultaneously. Unlike simple round-robin or least-connections load balancing, Dynamo's router understands the semantic content of incoming requests — specifically, which prefix tokens they share with cached entries — and factors this into routing decisions. The result is significantly higher effective cache hit rates compared to routing-unaware caching systems.

Planner and Scheduler for Dynamic Scaling

The planner is Dynamo's cluster-level resource manager. It monitors the state of all worker pools in real time — queue depths, GPU utilization, memory usage, request latency — and makes decisions about how to allocate GPU resources across the models being served on the cluster. In an AI factory environment where multiple models run simultaneously, the planner enables the cluster to behave as a single shared resource pool rather than a collection of fixed per-model allocations.

When demand for one model spikes — for example, a new product feature launches and traffic to a specific assistant endpoint increases tenfold — the planner can redirect GPUs from lower-utilization model pools to the high-demand pool without human intervention or manual reconfiguration. The reallocation is not instantaneous (workers need to load model shards), but the planner operates ahead of demand by monitoring queue growth trends and initiating reallocation before queues become critically long.

The scheduler operates at a lower level than the planner, making per-request assignment decisions within the current worker pool configuration. It integrates the KV cache locality information from the routing layer with real-time worker load to select the optimal worker for each request. The scheduler also handles the complexity of streaming output: in a disaggregated architecture, the decode worker that generates a response may be different from the prefill worker that processed the input, and the scheduler coordinates the KV state transfer between them transparently.

Multi-Framework and Multi-Model Support

One of the practical strengths of Dynamo's design is its pluggable backend architecture. Rather than building its own model execution engine, Dynamo integrates with the existing open-source inference engines that organizations are already using — vLLM, TensorRT-LLM, and SGLang — as worker backends. This means teams can adopt Dynamo incrementally: deploy the distributed coordination layer on top of existing vLLM workers without replacing the engine itself.

On the model side, Dynamo supports all major open-weight model families including Llama 4 (Scout, Maverick), Mistral and Mixtral, Qwen 2.5, DeepSeek V3 and R1, Falcon, and Phi-4. The OpenAI API-compatible endpoint layer means existing applications built against the OpenAI API can point to a Dynamo cluster with minimal configuration changes — a critical practical detail for organizations migrating from proprietary API usage to self-hosted infrastructure.

Open Source Ecosystem and Integrations

The Apache 2.0 license is a deliberate choice that reflects NVIDIA's strategic interest in expanding the ecosystem of organizations building on GPU infrastructure. By releasing Dynamo as fully open source, NVIDIA enables cloud providers, enterprises, and research institutions to adopt and extend the system without commercial restrictions, accelerating adoption and generating community contributions that improve the core product.

Performance Benchmarks and Throughput Gains

NVIDIA published benchmark results at GTC 2026 comparing Dynamo to standalone TensorRT-LLM and vLLM on representative production workloads. The results demonstrate that the gains from disaggregation and distributed KV caching are highly workload-dependent — the larger the average input prompt, the higher the proportion of repeated prefixes, and the higher the request concurrency, the more significant the throughput improvement.

It is important to contextualize these numbers. The peak improvements occur under conditions that favor disaggregation: high concurrency, long prompts, and high prefix repetition rates. For low-concurrency deployments serving a single model with short varied prompts, Dynamo's overhead may produce only modest gains or even slight latency increases compared to a well-tuned single-node vLLM setup. Dynamo is not a universal optimization — it is specifically designed for the AI factory deployment pattern where its architectural advantages are most pronounced.

Deployment Scenarios and AI Factories

Understanding which deployments benefit most from Dynamo helps organizations prioritize where to invest the additional operational complexity of a distributed inference runtime. The value proposition scales with cluster size, workload diversity, and request volume.

For organizations building AI-powered digital transformation initiatives, Dynamo represents the infrastructure layer that makes ambitious AI product economics viable at scale. The difference between serving one million requests per day at two cents per request and serving them at one cent — enabled by 2× throughput improvement — can be the difference between a profitable product and an unsustainable cost center. As AI becomes embedded in more business workflows and customer touchpoints, inference efficiency becomes a core competency rather than a technical detail.

Limitations and Production Considerations

Dynamo's architectural advantages come with meaningful operational complexity that organizations should evaluate carefully before committing to deployment. The disaggregated architecture that enables its performance gains also introduces new failure modes, increased network dependency, and higher minimum infrastructure requirements that make it overkill for many use cases.

The pragmatic guidance is that Dynamo is the right choice when inference cost and throughput have become a meaningful constraint and the organization has the engineering capacity to operate a distributed system. For teams at earlier stages, investing in well-tuned vLLM or TRT-LLM on a single multi-GPU node will deliver most of the performance needed at a fraction of the operational complexity. Dynamo becomes the natural next step when scale demands it.