AI PCs and NPUs in 2026: Can They Really Run Local AI?

An AI PC with a 40-plus TOPS NPU clears Microsoft’s Copilot+ bar, but the headline TOPS number tells you almost nothing about whether it can run a local LLM at a usable speed. NPUs are genuinely good at small, sustained on-device models and Windows AI features — they are not the way to run a 70B model. This guide separates the marketing from the silicon.

The confusion is everywhere: buyers see “45 TOPS” or “50 TOPS of AI performance” on a spec sheet and assume it means the laptop can run Llama or Mistral locally. It can run some AI workloads beautifully — and a large language model is usually not one of them, at least not on the NPU. The reason is a metric mismatch that almost no marketing page explains.

Below, we decode TOPS into real tokens per second, show why autoregressive LLM inference is bound by memory bandwidth rather than raw compute, map exactly which Copilot+ features run on the NPU versus which fall back to the iGPU or CPU, and give an honest buy-this-for-that decision matrix. Every spec is traceable to the silicon vendor or a named benchmark.

Microsoft defined the “Copilot+ PC” tier with a hardware floor: a neural processing unit rated at 40-plus TOPS, at least 16 GB of RAM, and a 256 GB SSD, all on Windows 11 24H2. That bar is what unlocks the on-device feature set — Windows Recall, Live Captions with real-time translation, Click to Do, and Windows Studio Effects — all running locally on the NPU rather than in the cloud.

Four silicon families clear that bar today. Their NPU ratings are close to one another and very far from a discrete GPU. Crucially, these are NPU-only numbers; some marketing slides quote a combined platform figure (Intel’s 120 TOPS, for example) that sums the NPU, GPU and CPU. The NPU alone is what runs the Windows AI features.

TOPS — trillions of operations per second — is a theoretical peak-compute figure. It is a fine way to compare two NPUs doing the same fixed-function job. It is a misleading way to predict large language model speed, for two compounding reasons.

First, the absolute gap is enormous. A discrete RTX 4090 advertises 1,300-plus TOPS; a Copilot+ NPU sits at 40-80 TOPS. That is roughly a 29× difference. If TOPS were the LLM metric, an AI PC would be a non-starter. But that framing is itself a trap — because, second, TOPS is not how LLM speed is measured at all.

This is the two-stage debunk at the heart of every AI PC purchase decision. Stage one: the NPU’s TOPS is tiny next to a GPU. Stage two: it does not matter, because TOPS is the wrong metric for both of them. What you actually want to know is how fast the chip can stream a model’s weights out of memory — and that is a bandwidth question, which the next section makes concrete. If you are weighing an AI PC against a Mac or a GPU box on cost, our local-AI hardware buyer’s guide by price bracket runs the same numbers across every platform.

Autoregressive token generation works one token at a time, and each step has to read the model’s weights out of memory. For single-stream decoding, the bottleneck is almost never compute — it is how fast those weights can be streamed. Throughput is bound by memory bandwidth divided by the model size in memory. A 70B model in 4-bit quantization occupies roughly 40 GB and needs substantial sustained bandwidth just to reach single-digit tokens per second. The NPU’s TOPS rating never enters this equation.

The table below makes that vivid. The decode ceiling column is computed only from memory bandwidth and model size — TOPS is deliberately left out, because it does not appear in the math. These are theoretical ceilings; real-world output lands below them once runtime overhead, the KV cache and prompt processing are included.

The math is simple and worth doing by hand. Decode ceiling ≈ memory bandwidth ÷ ~4.5 GB (an 8B model at 4-bit). At 135 GB/s the Snapdragon X Elite tops out near 30 tok/s; at 152 GB/s the X2 Elite reaches about 34. That is the punchline: the X2 Elite nearly doubles the NPU’s TOPS (45 to 80) yet lifts the bandwidth ceiling only about 13%, because bandwidth rose only 13%. TOPS climbed; real LLM headroom barely budged. The Ryzen AI Max+ 395, with roughly 256 GB/s from a 256-bit LPDDR5X-8000 bus, clears 57 tok/s on the same workload — not because its NPU is bigger, but because its memory is wider.

None of this means the NPU is useless — the opposite. It is a fixed-function accelerator built for small, sustained AI tasks at very low power, and it does that job better than a CPU or GPU could. The flagship example is Phi Silica, Microsoft’s on-device small language model, preinstalled on the NPU of every Copilot+ PC and based on Phi-3.5-mini.

That ~4B ceiling is the key. The NPU is brilliant at small language models and narrow vision and audio tasks — and small models are exactly where most on-device agent work belongs. Our small language model business guide covers why a 3-9B model like Phi or Qwen handles the majority of real-world steps, and the NPU is the most power-efficient place to run the smallest of them. A reviewer testing NPU acceleration captured the experience well.

That silent, low-power profile is the whole point of the NPU. The table below separates the Windows AI features that genuinely run on the NPU from the open LLMs that buyers assume run there but do not — the single most common misconception in this category.

Here is the detail that kills most “I’ll just run Llama on my NPU” plans before they start. As of mid-2026, the mainstream local-LLM runtimes — Ollama, llama.cpp and LM Studio — do not route workloads to the NPU at all. They run on the integrated GPU (via Vulkan, ROCm or Metal) or the CPU. Those tools load GGUF-format models; the NPU needs models hand-converted to ONNX and compiled for the vendor’s execution provider — QNN for Qualcomm, OpenVINO for Intel.

That conversion path exists and works, but it is a specialist, opt-in pipeline rather than a drop-in accelerator. Qualcomm’s AI Hub publishes 175-plus pre-optimized ONNX models validated for Snapdragon X Elite, and SDKs such as Nexa enable NPU acceleration for a curated set — Llama-3B, Phi4-mini, Qwen3-4B and an OmniNeural-4B multimodal model. But if you download a GGUF off Hugging Face and point Ollama at it, the NPU sits idle.

There is one platform that breaks the “AI PCs can’t run big models” rule — and it proves the point precisely because of how it does it. The AMD Ryzen AI Max+ 395 (Strix Halo) pairs 128 GB of LPDDR5X-8000 on a 256-bit bus with a 40-CU RDNA 3.5 integrated GPU. Up to 96 GB of that memory can be allocated to the GPU, which is enough to hold a 70B model.

Read that carefully: the Ryzen AI Max+ 395 runs 70B because of wide unified memory and a capable iGPU, not because of its 50 TOPS NPU. It is the honest answer to “can an AI PC run a big LLM?” — yes, if you buy a $2,000-plus workstation-class machine whose value is memory capacity, and you accept single-digit- to-low-double-digit tokens per second. If running larger models on a laptop is the goal, the Gemma 12B on a laptop guide and our DGX Spark vs M5 Max vs RTX 6000 comparison walk the same unified-memory-versus-discrete-GPU trade-off in depth.

Microsoft’s own platform direction is the strongest signal that the NPU-only framing was always too narrow. At Build 2025, Microsoft announced Windows AI Foundry, the evolution of the Windows Copilot Runtime — a unified layer that selects, optimizes, fine-tunes and deploys AI models across NPU, GPU and CPU, spanning AMD, Intel, NVIDIA and Qualcomm silicon, and integrating Foundry Local with open catalogs including Ollama and NVIDIA NIMs.

Underneath it, Windows ML automatically selects the correct execution provider — QNN for Qualcomm NPUs, OpenVINO for Intel NPUs — and falls back to the GPU or CPU when an NPU provider is unavailable. Developers no longer bundle execution providers by hand. The architecture quietly concedes that the right engine depends on the workload, not on a single 40 TOPS threshold.

That trajectory matters for buyers planning two or three years ahead. The NPU’s role is consolidating around what it does best — efficient, always-on small-model and feature inference — while the heavy LLM work migrates to whichever engine has the memory bandwidth to feed it. Expect future Windows AI features to lean on a hybrid NPU- plus-GPU split rather than the NPU alone, and expect the marketing to keep quoting TOPS long after the platform has stopped treating it as the deciding number.

Match the machine to the job. The mistake is buying an AI PC expecting a local LLM box, or skipping one because “NPUs can’t run LLMs” when the Windows AI features are exactly what you wanted. Four clear lanes.

For most teams, the answer is “both, for different reasons”: a Copilot+ AI PC for the laptop fleet’s battery life and Windows AI features, and a separate GPU or unified-memory machine where heavier local inference actually happens. If you are mapping local versus cloud economics across a fleet, our local AI versus cloud subscription ROI analysis frames the same trade-off in dollars, and our AI transformation engagements start with exactly this kind of hardware-and-workload mapping.