AI Agent Eval in 2026: Eight Frameworks, One Honest Comparison

AI agent evaluation frameworks reached an inflection point in May 2026: the category now spans five commercial platforms and three open-source standards, each solving a distinct slice of the production-readiness problem. The right choice depends on your team size, compliance tier, existing observability stack, and — following OpenAI's March 9 acquisition of Promptfoo — which model vendor you trust to grade your agents.

The stakes are no longer theoretical. Eval tooling is strategic infrastructure. Braintrust closed an $80M Series B at an $800M valuation in February 2026 with customers including Notion, Replit, Cloudflare, and Ramp. OpenAI acquired Promptfoo — the security and red-team eval leader used by more than 25% of Fortune 500 companies — for a reported $86M valuation. LangSmith, backed by LangChain's 80M monthly downloads, is the default starting point for LangGraph teams. The commercial eval market is consolidating around a handful of serious platforms.

This guide covers all eight frameworks: the five commercial platforms (LangSmith, Braintrust, Helicone, Phoenix by Arize, and Promptfoo), and the three open-source standards (OpenAI Evals, DeepEval v4.0.3, and Inspect AI v0.3.225 from the UK AI Security Institute). It includes a full 8-framework comparison matrix, a breakdown of where the SOC 2 cliff actually falls per platform, a 5-step custom eval design template that goes from production incidents to CI/CD regression gates, and an honest look at the vendor-objectivity question that the Promptfoo acquisition raised but most roundups have ignored. For the deeper observability context, see our agent observability 2026 guide to evals, traces, and cost.

The AI agent eval landscape matured enough in early 2026 to segment cleanly into two tiers. Commercial platforms provide managed infrastructure — trace ingestion, annotation queues, dataset versioning, SOC 2 certifications, and enterprise SLAs — in exchange for subscription fees. Open-source frameworks provide eval logic, metric libraries, and CI/CD integration that runs on your own infrastructure, with no data leaving your environment and no monthly bill for the framework itself.

The two tiers are not mutually exclusive. Many teams run DeepEval or OpenAI Evals for unit-level CI checks and LangSmith or Braintrust for production trace annotation and dataset management. The workflow pattern that emerging engineering teams are converging on — as described in our deep-dive on LangSmith, Langfuse, and Arize Phoenix observability platforms — is OSS for speed at the PR level and commercial for compliance and audit at the production level.

The five commercial platforms differ primarily on architecture philosophy. LangSmith is tightly integrated with the LangChain ecosystem and optimized for LangGraph agent tracing. Braintrust is dataset-first and model-agnostic, with sandboxed Python custom scorers that no other platform currently offers. Helicone is a proxy wrapper — it requires zero SDK changes and bolts eval capability onto your existing LLM calls. Phoenix (Arize) is OpenTelemetry-native, giving it the strongest portability story for teams that already instrument with OTel. Promptfoo is security and red-team focused, now integrated into the OpenAI Frontier infrastructure following the March 2026 acquisition.

The three OSS standards each fill a distinct gap. OpenAI Evals (18.5k GitHub stars, MIT) is a registry-based framework for reproducible benchmark-style evals — the closest analog to running a published benchmark against your agent. DeepEval (15.7k stars, Apache 2.0) is the pytest-native framework, wrapping G-Eval, DAG metrics, and a full agentic eval harness into a developer ergonomics that feels like a standard Python test suite. Inspect AI (2.1k stars, MIT, UK AISI) ships 200+ pre-built evaluations across providers and is the gold standard for public-sector, safety-critical, and multi-provider testing environments.

Each commercial platform has a distinct architectural identity that shapes everything downstream: what data it captures, how custom metrics are written, whether it supports on-prem deployment, and which compliance tier it can satisfy. The cost delta between entry and SOC 2 tiers is the sharpest selection signal — covered in full in §05, but surfaced here so it is present while reading platform descriptions.

LangSmith— Three published tiers: Developer (free, 5,000 base traces/month), Plus ($39/seat/month, 10,000 base traces), and Enterprise (custom). SOC 2 Type II certification was announced July 15, 2024; HIPAA and GDPR compliance are also certified. Enterprise ships self-hosted deployment via Docker Compose or Kubernetes, plus a hybrid mode that keeps the data plane in the customer's VPC. The plus-tier SOC 2 coverage is the most accessible entry point in the commercial field. LangSmith's natural home is LangChain and LangGraph teams — Harrison Chase, LangChain CEO, described the founding purpose as helping “ close the gap between prototype and production” by giving developers visibility into LLM context. For teams outside the LangChain ecosystem, LangSmith is usable but carries integration friction that Phoenix (OTel) or Braintrust (model-agnostic) does not.

Braintrust— Starter (free, 1 GB processed data, 10k scores, 14-day retention), Pro ($249/mo, SOC 2 Type II, custom topics and environments), Enterprise (custom, on-prem available). The architectural differentiator is sandboxed Python custom scorers — the only commercial platform that runs custom scorer code in an isolated environment, reducing the risk of scorer-side side effects and enabling arbitrary Python logic without trust issues. Braintrust's February 2026 Series B announcement at $800M valuation with ICONIQ as lead investor validates the dataset-driven, model-agnostic positioning. Ankur Goyal, CEO, on the Latent Space podcast: “If you embrace evaluation as the sort of core workflow in AI engineering, meaning every time you make a change, you evaluate it… then you're able to build much, much better AI software.”

Helicone— Hobby (free, 10,000 requests, 1 GB storage, 7-day retention), Pro ($79/mo), Team ($799/mo, SOC 2 Type II + HIPAA), Enterprise (custom, on-prem available). The proxy architecture is Helicone's defining trait: teams add a single base-URL change and Helicone captures every LLM call automatically, with no SDK changes or instrumentation work. Scores and datasets are bolt-on capabilities. CI/CD integration is possible via webhooks. The SOC 2 cliff here is steep — $79/mo to $799/mo is a 10x jump for compliance. See the full breakdown in our observability stack TCO calculator across LangSmith, Langfuse, and Helicone.

Phoenix (Arize) — Phoenix itself is free, open-source (Elastic License 2.0), and self-hosted by default: 9.8k GitHub stars, Python (arize-phoenix-evals) and TypeScript (@arizeai/phoenix-evals) packages. The Arize AX managed cloud adds tiers: AX Free (25k spans/mo, 1 GB, 15-day retention), AX Pro ($50/mo), AX Enterprise (custom, SOC 2 Type II + HIPAA). Phoenix v16.0.0 shipped May 21, 2026 with sandboxed Code Evaluators for composite scoring, embedding-based eval, and LLM-jury implementations executed server-side. The OTel-native architecture is its key portability advantage — any team already instrumented with OpenTelemetry can point traces at Phoenix without SDK lock-in. No other platform in this list matches that portability story. See our deep-dive on LangSmith, Langfuse, and Arize Phoenix observability platforms for the full OTel integration comparison.

Promptfoo— Community tier free (10k probes/month for red teaming); Enterprise and On-Premise pricing is custom. SOC 2 certified and ISO 27001 certified at the organization level. Promptfoo's GitHub repo carries 21.5k stars and an MIT license; founders Ian Webster and Michael D'Angelo have committed to keeping it open source post-acquisition: “ Promptfoo will remain open source and we will continue to serve users and customers.” The acquisition is documented in a March 9, 2026 TechCrunch report: Promptfoo had raised just $23M and was valued at $86M after its most recent round in July 2025. The vendor-objectivity concern this raises for non-OpenAI teams is addressed in full in §08.

Open-source eval frameworks ship no SOC 2, no managed trace storage, and no enterprise SLA. What they ship instead is eval logic you own entirely, metrics you can audit, and CI/CD integration that runs in any Python or Node environment. For teams at the prototype-to-early- production phase, these frameworks provide the fastest path to meaningful eval coverage. For regulated teams, they are the CI layer that sits under a commercial platform's managed data pipeline.

OpenAI Evals (18.5k GitHub stars, MIT, github.com/openai/evals) is a framework for evaluating LLMs and LLM systems, plus an open-source registry of benchmarks. It is template-driven: eval definitions live in YAML, custom code evals are supported, and the registry provides a library of community-contributed benchmarks. One important caveat: OpenAI Evals does not ship a CI/CD runner — it is a Python-runnable framework plus registry. CI integration is bring-your-own glue (a GitHub Actions step that calls the eval script). The repo has 691 commits on main as of May 2026 with no tagged GitHub releases; development cadence is irregular. Best fit: teams running reproducible benchmark-style evals and contributing to or consuming the community benchmark registry.

DeepEval v4.0.3 (15.7k GitHub stars, Apache 2.0, Confident AI, github.com/confident-ai/deepeval) released May 21, 2026 with Decision Graph Logic for granular simulation control. v4.0.2 (May 13, 2026) shipped a coding-agent eval harness and a terminal trace inspection TUI. The framework is fully pytest-integrated: deepeval test runin CI is the canonical command. Metrics cover the full agent lifecycle — G-Eval and DAG for LLM-as-judge, Answer Relevancy and Faithfulness for RAG, Task Completion and Tool Correctness for agents, plus Hallucination, Bias, and Toxicity as safety metrics. DeepEval is the closest open-source analog to a commercial eval platform's metric library, without the managed infrastructure. Confident AI offers a cloud layer for teams that want managed datasets and history. See our AI evaluation metrics reference guide for how G-Eval and DAG compare to deterministic scorers.

Inspect AI v0.3.225 (2.1k GitHub stars, MIT, github.com/UKGovernmentBEIS/inspect_ai) released May 23, 2026, maintained by the UK AI Security Institute and Meridian Labs. The defining feature is 200+ pre-built evals spanning OpenAI, Anthropic, Google, Grok, Mistral, Hugging Face, AWS Bedrock, Azure AI, vLLM, and Ollama — no other framework approaches that provider coverage out of the box. Releases are tracked via PyPI (not GitHub Releases). The official documentation site is the canonical reference. Best fit: public-sector teams with multi-provider requirements, safety-critical workloads, and teams that want to run established safety benchmarks without writing custom metric code.

No existing comparison in May 2026 covers all eight frameworks — commercial and open-source — in a single matrix with SOC 2 tier, on-prem availability, and CI/CD support explicit. Most roundups cover four commercial platforms or three open-source tools; the cross-tier view is absent. The matrix below is sourced from vendor pricing pages (retrieved 2026-05-24), GitHub repositories, and the research data confirmed in this post's fact file. For the deeper observability-first view of the commercial subset, see our 30/60/90-day observability rollout plan.

Every comparison post we surveyed in May 2026 lists pricing tiers without naming the exact moment when SOC 2 compliance kicks in. That gap matters for procurement decisions. The delta between a functional trial tier and a compliant production tier can be 10x — and for teams that need compliance on day one, the entry price is not the number that matters. The number that matters is the SOC 2 floor.

The analysis below is grounded in vendor pricing pages retrieved 2026-05-24. Pricing changes without notice; re-verify before procurement.

Three structural patterns emerge from this breakdown. First, LangSmith is the only commercial platform where SOC 2 compliance is accessible at a per-seat price — $39/seat means a team of five pays $195/month for a compliant tier. Braintrust's $249/mo is flat-rate (no per-seat cliff for small teams). Helicone's $799/mo Team tier is flat-rate but jumps from $79 — that delta is the highest absolute cliff in the group.

Second, OSS frameworks carry no SOC 2 of their own. Teams running DeepEval or Inspect AI self-hosted are responsible for their own infrastructure's compliance posture — SOC 2 for the eval layer means SOC 2 for the compute environment running it. This is not a disqualifier for OSS; it is a procurement framing point.

Third, on-prem availability is inconsistent in ways that matter for regulated workloads. LangSmith ships self-hosted plus a hybrid VPC mode on Enterprise. Braintrust ships on-prem on Enterprise. Helicone ships on-prem only on Enterprise. Arize ships Phoenix as the self-hosted product by default (it's the OSS). For regulated data environments where traces cannot leave the customer network, the on-prem availability tier is the dominant constraint — and most tier-comparison posts bury it. Our Agent Success Rate (ASR) methodology covers how to structure eval pipelines for regulated environments where external data transfer is restricted.

SWE-Bench Verified is the most-cited coding-agent benchmark in the eval ecosystem as of May 2026 — and one of the most consistently misread. The numbers most teams reference: Claude Opus 4.7 at 87.6% (Anthropic, April 16, 2026) and Codex CLI on GPT-5.5 at a reported 88.7%. Both are correct, and neither is comparable to the other in the way a head-to-head score comparison implies. Confirming Claude Opus 4.7's 87.6% SWE-Bench Verified score against the primary Anthropic announcement source is straightforward. The 88.7% figure for GPT-5.5 via Codex CLI is cross-referenced in our SWE-Bench Live leaderboard analysis — but comes with the same caveat.

The harness effect is the dominant variable. Identical model weights in different evaluation harnesses commonly produce 10-20 point score differences on SWE-Bench. The scoring depends on the task selection within the “Verified” subset, the scaffolding and tool access given to the model, how retries are counted, and how partial-credit patches are graded. When two vendors report different SWE-Bench numbers for their respective models, the harness is usually why — not the models themselves. “My model scored X on SWE-Bench Verified” is a different statement from “my model beats a competitor on SWE-Bench Verified” unless both used identical harness configurations.

One additional benchmark deserves mention for context. CursorBench, where Cursor's Composer 2.5 reportedly scores 63.2%, is built and scored by Cursor on Cursor's own harness — vendor-controlled methodology. Aider's polyglot benchmark is similarly self-scored. OSWorld (369 real computer tasks) and AgentBench (1,360 tasks across 8 environments) are academic, externally run benchmarks — a different validity tier from vendor-published numbers.

The practical implication for eval framework selection: before choosing a platform based on a model's published SWE-Bench score, establish your own internal benchmark on a golden set of tasks representative of your actual production workload. Published leaderboard positions tell you about the model in the benchmark vendor's harness. Your harness is the only number that matters for your production agent. This is the foundation of step 4 in the custom eval design template below — establish baseline + acceptance bar against your own golden dataset, not against a published benchmark. See our cost-per-successful-task as an eval metric for the economic framing that ties task completion rates to actual production value.

Every framework in this roundup ships documentation on how to write a metric. None ships a vendor-neutral guide on how to design a custom eval program from scratch — starting from production incidents and ending in a CI/CD gate that blocks deploys. That practitioner gap is what the 5-step template below fills.

The template applies to any framework combination: LangSmith for dataset storage + DeepEval for CI checks, Braintrust end-to-end, or Inspect AI for provider-agnostic coverage with self-hosted trace capture. The framework is the tool; the 5-step process is the methodology. For the related pass-rate and revision-rate framing, see our agent quality metrics — pass rate and revision rate guide, and for the prompt-level regression complement, see the prompt-library regression framework.

Every I/O 2026 and March 2026 news cycle produced pieces on the Promptfoo acquisition that essentially rewrote the press release: OpenAI bought Promptfoo, the tool remains open source, Ian Webster committed to vendor neutrality. Those facts are all correct. None of the pieces we surveyed raised the structural question that matters for enterprise procurement: if your security and red-team eval tooling is owned by a model vendor, is the grading structurally neutral for non-OpenAI model workflows?

The concern is not that Promptfoo will actively bias scores against Claude Opus 4.7, Gemini 3.5 Flash, or open-source models. The concern is structural. Promptfoo is integrated into “OpenAI Frontier” for “automated red-teaming, evaluating agentic workflows for security concerns, and monitoring activities for risks and compliance needs” (per Futurum's post-acquisition analysis). The development priorities, benchmark selection, and integration depth will now reflect OpenAI's product roadmap — not a neutral open-source foundation's roadmap.

For teams running Claude Opus 4.7 or Gemini 3.5 Flash in production, the red-team coverage depth for non-OpenAI model attack surfaces may lag behind OpenAI model coverage over time. This is not fabrication — it is the standard analysis applied to any eval tool with a dominant patron. Applied to CursorBench (built by Cursor, scored by Cursor) or Aider's polyglot benchmark (built by Aider), the same skepticism is warranted. Promptfoo's founders have explicitly committed to open-source maintenance. Take that commitment at face value and track whether the commit activity on non-OpenAI provider integrations diverges post-acquisition.

The practical guidance: if your production agents run on non-OpenAI models and red-team coverage for those specific models is a material compliance requirement, supplement Promptfoo with Inspect AI (which covers OpenAI, Anthropic, Google, Grok, Mistral, AWS Bedrock, and Azure AI in its 200+ pre-built evals) or run DeepEval's Hallucination and Bias metrics as an independent scoring layer. The goal is not to avoid Promptfoo — it is to not rely on a single vendor-controlled tool for your entire security eval surface. Our AI transformation advisory work includes eval-stack design for regulated teams with multi-vendor model portfolios precisely because of this kind of structural risk.

The eight-framework matrix is useful for inventory. The decision heuristic below converts that inventory into a starting point. Three variables drive the initial cut: team size and existing stack, the compliance tier required in the next 12 months, and whether eval needs to run inside the network or can send traces to a cloud provider.

Start on LangSmith ifyou are already on LangChain or LangGraph. The automatic trace capture, native LangGraph integration, and $39/seat SOC 2 entry make it the zero-friction starting point for that ecosystem. If you are not in the LangChain ecosystem, LangSmith's friction is real — evaluate Phoenix or Braintrust instead.

Start on Braintrust if you need dataset-driven eval with model-agnostic scoring and are willing to pay the $249/mo Pro floor for SOC 2. The sandboxed Python scorer capability is a genuine differentiator for teams that need custom business-logic metrics that cannot be expressed in template-based scorers. If you are not yet at the $249/mo commitment point, start with DeepEval OSS + a git-versioned golden dataset and migrate to Braintrust when the dataset management overhead justifies the subscription.

Start on Phoenix + DeepEval if you need self-hosted trace storage plus a pytest-native CI eval layer. Phoenix OSS handles trace capture and visualization with full OTel portability; DeepEval handles the CI eval run. Both are free and self-hostable. This combination covers the 80% use case for engineering teams that are not yet in a compliance-gated procurement cycle. For the broader observability stack design, see our agent observability 2026 guide to evals, traces, and cost.

Start on Inspect AI if you work in the public sector, a regulated health or finance environment, or need to run evals across five or more model providers. The 200+ pre-built evals and provider-agnostic design mean you can benchmark Claude Opus 4.7, GPT-5.5, Gemini 3.5 Flash, Mistral, and a Bedrock model in a single eval run without writing custom adapter code. The 2.1k GitHub star count understates its importance in the safety and public-sector ecosystem.

Add Promptfoo to your stack ifsecurity red-teaming and automated jailbreak testing are material requirements — and you have considered the vendor-objectivity implications for non-OpenAI model portfolios described in §08. Promptfoo's community tier (free, 10k probes/month) makes it the easiest red-team layer to add to any existing stack without procurement friction.

Use the build a Codex test-generation pipeline tutorial as the CI integration reference if you are wiring any of these frameworks into a GitHub Actions workflow. The patterns apply across DeepEval, LangSmith, and Promptfoo with minimal adaptation.