AI Agent Eval in 2026: Eight Frameworks, One Honest Comparison

AI agent evaluation frameworks reached an inflection point in May 2026: the category now spans five commercial platforms and three open-source standards, each solving a distinct slice of the production-readiness problem. The right choice depends on your team size, compliance tier, existing observability stack, and — following OpenAI's March 9 acquisition of Promptfoo — which model vendor you trust to grade your agents.

Braintrust closed an $80M Series B at an $800M valuation in February 2026 with customers including Notion, Replit, Cloudflare, and Ramp. OpenAI acquired Promptfoo — the security and red-team eval tool used by more than 25% of Fortune 500 companies — for a reported $86M valuation. LangSmith, backed by LangChain's 80M monthly downloads, is the default starting point for LangGraph teams.

This guide covers all eight frameworks: the five commercial platforms (LangSmith, Braintrust, Helicone, Phoenix by Arize, and Promptfoo), and the three open-source standards (OpenAI Evals, DeepEval v4.0.3, and Inspect AI v0.3.225 from the UK AI Security Institute). It includes a comparison matrix, where SOC 2 sits in each vendor's pricing, a 5-step custom eval design template with a worked regression gate, and the vendor-objectivity question the Promptfoo acquisition raised. For the deeper observability context, see our agent observability 2026 guide to evals, traces, and cost.

The AI agent eval landscape matured enough in early 2026 to segment cleanly into two tiers. Commercial platforms provide managed infrastructure — trace ingestion, annotation queues, dataset versioning, SOC 2 certifications, and enterprise SLAs — in exchange for subscription fees. Open-source frameworks provide eval logic, metric libraries, and CI/CD integration that runs on your own infrastructure, with no data leaving your environment and no monthly bill for the framework itself.

The two tiers are not mutually exclusive. Many teams run DeepEval or OpenAI Evals for unit-level CI checks and LangSmith or Braintrust for production trace annotation and dataset management. The workflow pattern that emerging engineering teams are converging on — as described in our deep-dive on LangSmith, Langfuse, and Arize Phoenix observability platforms — is OSS for speed at the PR level and commercial for compliance and audit at the production level.

The five commercial platforms differ primarily on architecture philosophy. LangSmith is tightly integrated with the LangChain ecosystem and optimized for LangGraph agent tracing. Braintrust is dataset-first and model-agnostic, with sandboxed Python custom scorers. Helicone is a proxy wrapper — it requires zero SDK changes and bolts eval capability onto your existing LLM calls. Phoenix (Arize) is OpenTelemetry-native, giving it the strongest portability story for teams that already instrument with OTel. Promptfoo is security and red-team focused, now integrated into the OpenAI Frontier infrastructure following the March 2026 acquisition.

The three OSS standards each fill a distinct gap. OpenAI Evals (18.5k GitHub stars, MIT) is a registry-based framework for reproducible benchmark-style evals — the closest analog to running a published benchmark against your agent. DeepEval (15.7k stars, Apache 2.0) is the pytest-native framework, wrapping G-Eval, DAG metrics, and a full agentic eval harness into a developer ergonomics that feels like a standard Python test suite. Inspect AI (2.1k stars, MIT, UK AISI) ships 200+ pre-built evaluations across providers and is the gold standard for public-sector, safety-critical, and multi-provider testing environments.

Each commercial platform's architecture decides what data it captures, how custom metrics are written, whether it supports on-prem deployment, and which compliance tier it can satisfy. Where SOC 2 sits in each price list is covered in §05. Tier names and limits below are from the May 24, 2026 retrieval; the SOC 2 tiers were rechecked on October 3, 2026.

LangSmith — Three published tiers: Developer (free, 5,000 base traces/month), Plus ($39/seat/month, 10,000 base traces), and Enterprise (custom). SOC 2 Type II certification was announced July 15, 2024; HIPAA and GDPR compliance are also certified. Enterprise ships self-hosted deployment via Docker Compose or Kubernetes, plus a hybrid mode that keeps the data plane in the customer's VPC. LangSmith's shared-responsibility docs state SOC 2 Type II, GDPR, and HIPAA compliance for the platform, and the pricing page (checked October 3, 2026) does not tie SOC 2 to a plan. LangSmith's natural home is LangChain and LangGraph teams — Harrison Chase, LangChain CEO, described the founding purpose as helping “close the gap between prototype and production” by giving developers visibility into LLM context. For teams outside the LangChain ecosystem, LangSmith is usable but carries integration friction that Phoenix (OTel) or Braintrust (model-agnostic) does not.

Braintrust — Starter (free, 1 GB processed data, 10k scores, 14-day retention), Pro ($249/mo, custom charts and environments), Enterprise (custom, on-prem available). The pricing page checked October 3, 2026 lists SOC 2 Type II on all three plans, Starter included; the HIPAA business associate agreement is Enterprise-only. The architectural differentiator is sandboxed Python custom scorers, which run custom scorer code in an isolated environment, reducing the risk of scorer-side side effects and enabling arbitrary Python logic without trust issues. Braintrust's February 2026 Series B announcement at $800M valuation with ICONIQ as lead investor validates the dataset-driven, model-agnostic positioning. Ankur Goyal, CEO, on the Latent Space podcast: “If you embrace evaluation as the sort of core workflow in AI engineering, meaning every time you make a change, you evaluate it… then you’re able to build much, much better AI software.”

Helicone — Hobby (free, 10,000 requests, 1 GB storage, 7-day retention), Pro ($79/mo), Team ($799/mo, SOC 2 Type II + HIPAA), Enterprise (custom, on-prem available). The proxy architecture is Helicone's defining trait: teams add a single base-URL change and Helicone captures every LLM call automatically, with no SDK changes or instrumentation work. Scores and datasets are bolt-on capabilities. CI/CD integration is possible via webhooks. The SOC 2 cliff here is steep — $79/mo to $799/mo is a 10x jump for compliance. See the full breakdown in our observability stack TCO calculator across LangSmith, Langfuse, and Helicone.

Phoenix (Arize) — Phoenix itself is free, open-source (Elastic License 2.0), and self-hosted by default: 9.8k GitHub stars, Python (arize-phoenix-evals) and TypeScript (@arizeai/phoenix-evals) packages. The Arize AX managed cloud adds tiers: AX Free (25k spans/mo, 1 GB, 15-day retention), AX Pro ($50/mo, 50k spans/mo), AX Enterprise (custom). Arize's pricing page, checked October 3, 2026, lists SOC 2 Type II from AX Pro and HIPAA on Enterprise only. Phoenix v16.0.0 shipped May 21, 2026 with sandboxed Code Evaluators for composite scoring, embedding-based eval, and LLM-jury implementations executed server-side. The OTel-native architecture is its key portability advantage — any team already instrumented with OpenTelemetry can point traces at Phoenix without SDK lock-in. See our deep-dive on LangSmith, Langfuse, and Arize Phoenix observability platforms for the full OTel integration comparison.

Promptfoo — Community tier free (10k probes/month for red teaming); Enterprise and On-Premise pricing is custom. SOC 2 certified and ISO 27001 certified at the organization level. Promptfoo's GitHub repo carries 21.5k stars and an MIT license; founders Ian Webster and Michael D'Angelo have committed to keeping it open source post-acquisition: “Promptfoo will remain open source and we will continue to serve users and customers.” The acquisition is documented in a March 9, 2026 TechCrunch report: Promptfoo had raised just $23M and was valued at $86M after its most recent round in July 2025. The vendor-objectivity concern this raises for non-OpenAI teams is addressed in full in §08.

Open-source eval frameworks ship no SOC 2, no managed trace storage, and no enterprise SLA. What they ship instead is eval logic you own entirely, metrics you can audit, and CI/CD integration that runs in any Python or Node environment. For teams at the prototype-to-early- production phase, these frameworks provide the fastest path to meaningful eval coverage. For regulated teams, they are the CI layer that sits under a commercial platform's managed data pipeline.

OpenAI Evals (18.5k GitHub stars, MIT, github.com/openai/evals) is a framework for evaluating LLMs and LLM systems, plus an open-source registry of benchmarks. It is template-driven: eval definitions live in YAML, custom code evals are supported, and the registry provides a library of community-contributed benchmarks. One important caveat: OpenAI Evals does not ship a CI/CD runner — it is a Python-runnable framework plus registry. CI integration is bring-your-own glue (a GitHub Actions step that calls the eval script). The repo has 691 commits on main as of May 2026 with no tagged GitHub releases; development cadence is irregular. Best fit: teams running reproducible benchmark-style evals and contributing to or consuming the community benchmark registry.

DeepEval v4.0.3 (15.7k GitHub stars, Apache 2.0, Confident AI, github.com/confident-ai/deepeval) released May 21, 2026 with Decision Graph Logic for granular simulation control. v4.0.2 (May 13, 2026) shipped a coding-agent eval harness and a terminal trace inspection TUI. The framework is fully pytest-integrated: deepeval test run in CI is the canonical command. Metrics cover the full agent lifecycle — G-Eval and DAG for LLM-as-judge, Answer Relevancy and Faithfulness for RAG, Task Completion and Tool Correctness for agents, plus Hallucination, Bias, and Toxicity as safety metrics. DeepEval is the closest open-source analog to a commercial eval platform's metric library, without the managed infrastructure. Confident AI offers a cloud layer for teams that want managed datasets and history. See our AI evaluation metrics reference guide for how G-Eval and DAG compare to deterministic scorers.

Inspect AI v0.3.225 (2.1k GitHub stars, MIT, github.com/UKGovernmentBEIS/inspect_ai) released May 23, 2026, maintained by the UK AI Security Institute and Meridian Labs. The defining feature is 200+ pre-built evals spanning OpenAI, Anthropic, Google, Grok, Mistral, Hugging Face, AWS Bedrock, Azure AI, vLLM, and Ollama — no other framework approaches that provider coverage out of the box. Releases are tracked via PyPI (not GitHub Releases). The official documentation site is the canonical reference. Best fit: public-sector teams with multi-provider requirements, safety-critical workloads, and teams that want to run established safety benchmarks without writing custom metric code.

The matrix puts commercial and open-source frameworks side by side with SOC 2 tier, on-prem availability, and CI/CD support stated for each. It is sourced from vendor pricing pages (retrieved 2026-05-24; SOC 2 tiers rechecked 2026-10-03) and the GitHub repositories. For the deeper observability-first view of the commercial subset, see our 30/60/90-day observability rollout plan.

Pricing roundups rarely say at which tier SOC 2 starts, and for a team that needs compliance on day one, that tier is the price that matters. The gap between a trial tier and the first compliant tier can be 10x.

The chart reflects vendor pricing pages and docs checked 2026-10-03, and it has moved since our May 24 retrieval: Braintrust then reserved SOC 2 for the $249/mo Pro tier and now lists it on the free Starter plan, and Arize then reserved it for AX Enterprise and now lists it from AX Pro at $50/mo. Pricing changes without notice; re-verify before procurement.

Three patterns follow. First, SOC 2 has largely stopped being a paywall: of the four vendors we rechecked, three now state it for a free or $50 tier. Helicone is the exception, and its $79 to $799 step is the cliff to price in. A SOC 2 listing is not the whole compliance story, though: HIPAA still sits on Enterprise at Braintrust and Arize and on Team at Helicone, so a health-data workload still meets a price step.

Second, OSS frameworks carry no SOC 2 of their own. Teams running DeepEval or Inspect AI self-hosted are responsible for their own infrastructure's compliance posture — SOC 2 for the eval layer means SOC 2 for the compute environment running it. This is not a disqualifier for OSS; it is a procurement framing point.

Third, on-prem availability is inconsistent in ways that matter for regulated workloads. LangSmith ships self-hosted plus a hybrid VPC mode on Enterprise. Braintrust ships on-prem on Enterprise. Helicone ships on-prem only on Enterprise. Arize ships Phoenix as the self-hosted product by default (it's the OSS). For regulated data environments where traces cannot leave the customer network, the on-prem availability tier is the dominant constraint. Our Agent Success Rate (ASR) methodology covers how to structure eval pipelines for regulated environments where external data transfer is restricted.

SWE-Bench Verified is the most-cited coding-agent benchmark in the eval ecosystem as of May 2026 — and one of the most consistently misread. The numbers most teams reference: Claude Opus 4.7 at 87.6% (Anthropic, April 16, 2026) and Codex CLI on GPT-5.5 at a reported 88.7%. Both are correct, and neither is comparable to the other in the way a head-to-head score comparison implies. Confirming Claude Opus 4.7's 87.6% SWE-Bench Verified score against the primary Anthropic announcement source is straightforward. The 88.7% figure for GPT-5.5 via Codex CLI is cross-referenced in our SWE-Bench Live leaderboard analysis — but comes with the same caveat.

The harness effect is the dominant variable. Identical model weights in different evaluation harnesses commonly produce 10-20 point score differences on SWE-Bench. The scoring depends on the task selection within the “Verified” subset, the scaffolding and tool access given to the model, how retries are counted, and how partial-credit patches are graded. When two vendors report different SWE-Bench numbers for their respective models, the harness is usually why — not the models themselves. “My model scored X on SWE-Bench Verified” is a different statement from “my model beats a competitor on SWE-Bench Verified” unless both used identical harness configurations.

One additional benchmark deserves mention for context. CursorBench, where Cursor's Composer 2.5 reportedly scores 63.2%, is built and scored by Cursor on Cursor's own harness — vendor-controlled methodology. Aider's polyglot benchmark is similarly self-scored. OSWorld (369 real computer tasks) and AgentBench (1,360 tasks across 8 environments) are academic, externally run benchmarks — a different validity tier from vendor-published numbers.

The practical implication for eval framework selection: before choosing a platform based on a model's published SWE-Bench score, establish your own internal benchmark on a golden set of tasks representative of your actual production workload. Published leaderboard positions tell you about the model in the benchmark vendor's harness. Your harness is the only number that matters for your production agent. This is the foundation of step 4 in the custom eval design template below — establish baseline + acceptance bar against your own golden dataset, not against a published benchmark. See our cost-per-successful-task as an eval metric for the economic framing that ties task completion rates to actual production value.

Framework documentation explains how to write a metric. The 5-step template below covers the part around it: how to go from production incidents to a CI/CD gate that blocks deploys, independent of which vendor stores the results.

The template applies to any framework combination: LangSmith for dataset storage + DeepEval for CI checks, Braintrust end-to-end, or Inspect AI for provider-agnostic coverage with self-hosted trace capture. For the related pass-rate and revision-rate framing, see our agent quality metrics — pass rate and revision rate guide, and for the prompt-level regression complement, see the prompt-library regression framework.

Worked example: one prompt change through the gate

Take a customer-support agent whose incident log produced three failure modes: calling the wrong tool, stating an order status the order record does not support, and repeating a customer's personal data back in a reply. The team labels 80, 60 and 40 golden cases for them, 180 in all. These inputs are illustrative, not client data; the arithmetic is exact.

The gate has two rules, written before anyone scores a candidate. No failure mode may lose more than one case against the production baseline, and the personal-data mode is a hard bar: every case must pass. Tolerance is set in cases, not percentage points, because one case is worth 1.25 points out of 80, 1.67 points out of 60 and 2.5 points out of 40. A “within 2 points” alert band on the 40-case mode is narrower than a single case, so it either never fires or fires on every flip.

The aggregate rose from 89.4% to 90.6%, and a dashboard that showed only that number would have shipped the change. The per-mode view shows the trade: the new prompt calls the order-lookup tool more often, then paraphrases the result into statuses the record does not contain. That is the regression the incident log was built to catch, and the gate blocks the merge on it.

Two checks keep a gate like this honest. First, score the unchanged production agent two or three times before trusting the tolerance. If the baseline itself moves by two cases between runs on the 60-case mode, a one-case tolerance will block good changes at random; the fix is more golden cases for that mode, not a looser rule. Second, grade the order-status mode with a deterministic check where you can, comparing the stated status with the record the tool returned, and keep the LLM judge for tone and wording. Anthropic's evaluation guide ranks code-based grading as the fastest and most reliable method and advises testing an LLM grader's reliability before scaling it.

Most March 2026 coverage of the Promptfoo acquisition restated the announcement: OpenAI bought Promptfoo, the tool remains open source, Ian Webster committed to vendor neutrality. Those facts are correct. The question that matters for enterprise procurement is a different one: if your security and red-team eval tooling is owned by a model vendor, is the grading structurally neutral for non-OpenAI model workflows?

The concern is not that Promptfoo will actively bias scores against Claude Opus 4.7, Gemini 3.5 Flash, or open-source models. The concern is structural. Promptfoo is integrated into “OpenAI Frontier” for “automated red-teaming, evaluating agentic workflows for security concerns, and monitoring activities for risks and compliance needs” (per Futurum's post-acquisition analysis). The development priorities, benchmark selection, and integration depth will now reflect OpenAI's product roadmap — not a neutral open-source foundation's roadmap.

For teams running Claude Opus 4.7 or Gemini 3.5 Flash in production, the red-team coverage depth for non-OpenAI model attack surfaces may lag behind OpenAI model coverage over time. That is the standard analysis for any eval tool with a dominant patron. Applied to CursorBench (built by Cursor, scored by Cursor) or Aider's polyglot benchmark (built by Aider), the same skepticism is warranted. Promptfoo's founders have explicitly committed to open-source maintenance. Take that commitment at face value and track whether the commit activity on non-OpenAI provider integrations diverges post-acquisition.

The practical guidance: if your production agents run on non-OpenAI models and red-team coverage for those specific models is a material compliance requirement, supplement Promptfoo with Inspect AI (which covers OpenAI, Anthropic, Google, Grok, Mistral, AWS Bedrock, and Azure AI in its 200+ pre-built evals) or run DeepEval's Hallucination and Bias metrics as an independent scoring layer. The goal is not to avoid Promptfoo — it is to not rely on a single vendor-controlled tool for your entire security eval surface. Our AI transformation advisory work includes eval-stack design for regulated teams with multi-vendor model portfolios precisely because of this kind of structural risk.

Three variables drive the initial cut: team size and existing stack, the compliance tier required in the next 12 months, and whether eval needs to run inside the network or can send traces to a cloud provider.

Start on LangSmith if you are already on LangChain or LangGraph. The automatic trace capture, native LangGraph integration, and platform-wide SOC 2 make it the low-friction starting point for that ecosystem. If you are not in the LangChain ecosystem, LangSmith's friction is real — evaluate Phoenix or Braintrust instead.

Start on Braintrust if you need dataset-driven eval with model-agnostic scoring. SOC 2 is listed on the free Starter plan, so a compliance requirement no longer forces the $249/mo Pro tier; Starter's 14-day retention and 10k scores usually do. The sandboxed Python scorer capability suits teams that need custom business-logic metrics that cannot be expressed in template-based scorers.

Start on Phoenix + DeepEval if you need self-hosted trace storage plus a pytest-native CI eval layer. Phoenix OSS handles trace capture and visualization with full OTel portability; DeepEval handles the CI eval run. Both are free and self-hostable. This combination covers most needs for engineering teams that are not yet in a compliance-gated procurement cycle. For the broader observability stack design, see our agent observability 2026 guide to evals, traces, and cost.

Start on Inspect AI if you work in the public sector, a regulated health or finance environment, or need to run evals across five or more model providers. The 200+ pre-built evals and provider-agnostic design mean you can benchmark Claude Opus 4.7, GPT-5.5, Gemini 3.5 Flash, Mistral, and a Bedrock model in a single eval run without writing custom adapter code. The 2.1k GitHub star count understates its importance in the safety and public-sector ecosystem.

Add Promptfoo to your stack if security red-teaming and automated jailbreak testing are material requirements — and you have considered the vendor-objectivity implications for non-OpenAI model portfolios described in §08. Promptfoo's community tier (free, 10k probes/month) makes it the easiest red-team layer to add to any existing stack without procurement friction.

Use the build a Codex test-generation pipeline tutorial as the CI integration reference if you are wiring any of these frameworks into a GitHub Actions workflow. The patterns apply across DeepEval, LangSmith, and Promptfoo with minimal adaptation.