LLM Gateway Architecture: The 2026 Engineering Reference

An LLM gateway is the proxy layer that sits between your application and the model providers you call — and in 2026 it has graduated from a convenience to critical AI infrastructure. It is where unified API access, caching, routing, fallbacks, budget enforcement, and compliance logging all live, so that none of those concerns leak into application code.

The shift is structural. Cloudflare's internal analysis suggests the average enterprise engineering team now calls several different model providers, which makes a single, vendor-neutral abstraction the default rather than the exception. Once you are routing to more than one provider — or spending real money on tokens — a gateway stops being optional and becomes the place where cost, reliability, and governance are actually enforced.

This reference compares the six most-deployed options — LiteLLM Proxy, Portkey, Cloudflare AI Gateway, Vercel AI Gateway, OpenRouter, and Kong AI Gateway — across the dimensions that actually decide an architecture: caching mechanics, routing algorithms, resilience patterns, governance and compliance controls, and the build-vs-buy economics most teams never quantify. Every figure below is sourced from official documentation, with vendor-stated claims flagged as such.

A gateway presents one API to your application and translates it into calls to whichever providers you have configured. That single translation point is what makes everything else possible: because all traffic flows through it, the gateway is the natural home for cross-cutting concerns that would otherwise be scattered across services. Vercel's documentation frames the role plainly — it gives you the ability to set budgets, monitor usage, load-balance requests, and manage fallbacks behind one endpoint.

The five responsibilities that define a production gateway are unified access, caching, routing, resilience, and governance. Each is examined in its own section below; the grid here is the mental map for how they relate.

Most public comparisons cover two or three tools and skip the dimensions that matter for compliance and cost. The table below assembles the six most-deployed gateways against the features that decide a real architecture. Every cell is drawn from each vendor's official documentation; the Kong row carries a caveat because its performance claims come from a vendor-authored benchmark (see the resilience section).

The matrix makes the real division visible: the choice is rarely about a missing feature. LiteLLM and Portkey win on data residency because everything runs on your infrastructure; Cloudflare and Vercel win on operational simplicity because they run the edge for you. OpenRouter wins on time-to-many-models. There is no gateway that is strictly best — there is the gateway that matches your residency requirements, your ops capacity, and your spend.

Caching is the single fastest ROI lever a gateway offers. A cached response can return in under 5 milliseconds, versus the 2 to 5 seconds a live inference call typically takes. Cost follows latency: a served-from-cache request costs nothing in provider tokens. Reported hit rates in the 30–40% range are a typical range cited in secondary analyses rather than a guaranteed number — but even a moderate hit rate compounds into meaningful savings at production scale.

There are two distinct mechanisms. Exact-match caching hashes the request and returns a stored response when an identical request arrives. Semantic caching goes further: it embeds the prompt and runs a cosine-similarity search against stored embeddings, returning a cached answer when a new prompt is close enough to a previous one. Portkey's implementation checks an exact hash first and falls back to vector similarity — a dual-layer design that captures both literal repeats and paraphrases.

The cosine-similarity threshold is the underexplored design parameter. Practitioners typically tune it in the 0.90 to 0.98 range, and the choice is a precision/recall trade-off with real consequences. Set it loose (near 0.90) and you risk false positives — returning a cached answer to a question that only superficially resembles the original. Set it tight (near 0.98) and the hit rate collapses toward exact-match levels. The gateway is the right place to host this cache precisely because it is shared: every service behind it benefits from one another's query history rather than each maintaining a siloed cache.

Routing decides which deployment serves each request. LiteLLM exposes a menu of strategies: Simple-Shuffle is the default and the one its docs recommend for production; Latency-Based routes to the fastest responder; Usage-Based-Routing-v2 (Redis-backed) routes to the deployment with the lowest tokens-per-minute load; Least-Busy picks the fewest concurrent requests; and Cost-Based optimizes for price. The strategy you pick encodes what you are optimizing for — and that should be an explicit decision, not a default left untouched.

OpenRouter takes a different, automatic approach. Its load balancer first prioritizes providers with no outages in the last 30 seconds, then weights the remaining candidates by the inverse square of their price — so a provider charging three times more is roughly nine times less likely to be selected. Explicit sort or order parameters disable this and force a sequential ordering instead. The same gateway implements the same sliding- and fixed-window rate limiting at the gateway layer that protects upstream providers from quota exhaustion.

Resilience is where the gateway earns its keep during an incident. Three layers stack. Retries handle transient failures: Portkey supports up to five attempts with exponential backoff. Fallbacks reroute to an alternate provider when retries won't help — and critically, there are three distinct fallback categories that most tutorials collapse into one. Circuit breakers sit on top, isolating a provider that is persistently failing so the system stops hammering it. These are the same retry and idempotency patterns that govern any reliable distributed system, applied at the provider-request level.

The circuit breaker deserves its own mental model. The right design keeps a separate breaker per provider+model combination — so an outage on one model doesn't block requests to a different model on the same provider. While the breaker is open during cooldown, it prevents retry storms; half-open probes then check periodically whether the provider has recovered. LiteLLM's cooldown mechanism, which trips when a deployment exceeds a 50% failure rate or returns 429s, is a circuit-breaker implementation in everything but name.

Governance is the dimension that turns a gateway from a performance tool into a compliance instrument. Budget enforcement works through request-level wallet checks: if the estimated cost of a request exceeds the available balance, the gateway rejects it with an HTTP 402 (Insufficient Balance) before it ever reaches the provider. A softer variant silently caps max_tokens to whatever the remaining balance can cover. Either way, the spend ceiling is enforced at the edge of your system, not discovered on an invoice. Per-user token attribution at this layer is the operational counterpart to the token cost attribution vocabulary every finance-aware AI team now needs.

Data-residency controls are equally a gateway concern. Vercel AI Gateway offers Zero Data Retention (ZDR) routing that sends requests only to providers contractually committed not to retain or train on prompt data — a compliance control enforced without touching application code. OpenRouter exposes ZDR as a provider filter. Cloudflare layers in Data Loss Prevention scanning for PII, financial, and healthcare data, plus real-time guardrails on both prompts and responses. The common thread: the gateway is where you enforce what data is allowed to leave your perimeter.

The single most consequential decision is self-hosted versus managed, and most teams decide it on instinct rather than arithmetic. OpenRouter gives the cleanest public benchmark for the managed layer: a 5.5% platform fee on credit purchases. (Note the boundary — that fee applies to credit purchases, not to bring-your-own-key usage; routing through BYOK avoids it.) That single number lets you build the crossover formula.

Self-hosted total cost is provider token spend plus infrastructure (compute, storage) plus observability tooling plus engineering hours at your loaded hourly rate. Managed total cost is provider token spend plus the platform fee. The crossover is the point where avoided engineering effort outweighs the fee. Work a concrete example: a team spending around $2,000/month on tokens pays roughly $110/month at a 5.5% managed fee. If self-hosting that gateway consumes even ten hours a month of maintenance at $150/hour, that's $1,500 in engineering cost — and the managed fee wins by an order of magnitude. The calculus flips only at high spend, where the percentage fee on a large token bill eventually exceeds a fixed engineering allocation.

Read that chart as illustrative, not prescriptive — the inputs are your numbers, not ours. The point is the method: the 5.5% fee is small in absolute terms until your token spend is large, while engineering time is expensive whether or not anything breaks. For most teams at moderate spend, the managed fee is the cheaper option once real engineering cost is counted. The case for self-hosting is rarely cost at moderate scale; it is data residency, regulatory mandate, or spend high enough that the percentage fee dominates. Decide on that axis, then let the arithmetic confirm it.

The decision collapses to four common situations. Match yours to the row below, then validate the choice against your own traffic — never against a headline.

Whichever you pick, treat the gateway as a load-bearing component from day one: instrument its logs for the compliance regime you answer to, tune your cache threshold against real traffic, and define your fallback categories explicitly. If you're standing up a production LLM stack and want the routing, caching, and governance architected correctly the first time, our AI transformation engagements start with exactly this kind of infrastructure decision — and our web and platform development team wires the gateway into your application end to end.