Tool-Calling Benchmarks Explained: BFCL, τ²-Bench and MCP

"Tool use" in a model launch post is one phrase standing in for at least five different tests, and the number that follows it can come from any of them. As of November 5, 2026 the Berkeley Function Calling Leaderboard grades whether a model emits a correctly formed call; τ²-bench grades whether an agent follows a written policy through a conversation with a simulated user; MCP-Universe, MCPMark and Toolathlon grade whether real servers end in the correct state after a multi-step task. A model can lead one and sit mid-table on another. This page sets out what each measures, how it scores, what its authors say is wrong with it, and who was on top when we read the boards.

Two boundaries apply throughout. We have not run any of these benchmarks and make no first-person claims about any model; every figure is copied from a public leaderboard or a paper, with the date that page displayed. And leaderboards move, so each table carries its own as-of stamp and the methodology section states the collection date once. For the general problem of reading a leaderboard, our benchmark methodology guide covers contamination and harness effects across all task types; this page is the tool-calling chapter.

The simplest way to keep the benchmarks apart is by what a task looks like and what counts as success. In the first family a task is a function schema and a request, and success is emitting the right call. In the second a task is a customer problem with a policy document, and success is the database ending in the annotated goal state after a conversation with a simulated user. In the third a task is an instruction over real software, such as a GitHub repository, a Notion workspace or a Kubernetes cluster, and success is a verification script passing against the final state of that software.

Each family has an obvious failure case the others do not test. A model that formats calls perfectly can still abandon a policy when a simulated customer pushes back. A model that handles a scripted airline conversation can still lose its place twenty turns into a job that touches four applications. And a model that completes long tasks on real servers may do so with a harness that retries and summarises on its behalf. The sections below take each benchmark in turn; the grid is the map.

The Berkeley Function Calling Leaderboard, built by the Gorilla team at UC Berkeley, is the benchmark named for "function calling". Its history is visible on the leaderboard page: V1 introduced abstract syntax tree matching as the metric, V2 added enterprise and community-contributed functions, V3 added multi-turn conversations, and V4 describes itself as holistic agentic evaluation, adding web search and memory. The V4 web-search release note, dated July 17, 2025, shows the composition at that point: 5,088 scored cases across agentic (665), multi-turn (800), live (1,351) and non-live (1,150) categories plus 1,122 hallucination-measurement cases, with a 5,200-case format-sensitivity set that does not score.

A single-turn task gives the model a function schema and a request, and the emitted call, name and arguments, is checked structurally against a set of acceptable answers. A web-search task gives the model a search tool and a multi-hop question; the release note's own example asks for the current mayor of the most populous city of the country with the most titles in a particular badminton championship. The scoring rule is where the published documents disagree with each other. The July 2025 note describes the overall score as a weighted average with agentic tasks at 40% and multi-turn at 30%; the leaderboard page as read on November 5, 2026 says overall accuracy is the unweighted average of all sub-categories. We cannot reconcile the two from the pages and so do not state the weights as current.

The authors' stated weakness is contamination by knowledge cutoff. When they re-ran the web-search set with search disabled, accuracy dropped sharply, but a few recent models still answered some questions from memory because the answers fell inside their training data, and the note says the question sets will be updated periodically to address it. Two structural caveats follow from the data file: the same model appears twice when it is run in native function-calling mode and in prompt mode, with different scores, and the board itself states it was last updated April 12, 2026, so nothing released after that date is on it.

τ-bench, from Sierra Research with Princeton co-authors, was submitted in June 2024 by Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan. Its design is a language-model-simulated user talking to an agent that has domain API tools and a policy document, in retail and airline settings. Success is judged by comparing the database state at the end of the conversation with an annotated goal state. The paper introduced pass^k, a reliability measure over repeated trials, and reported that GPT-4o-class agents succeeded on fewer than half of tasks, with pass^8 below 25% in retail. That reliability framing is the benchmark's reason to exist.

τ²-bench, submitted June 2025 by Victor Barres and colleagues, adds a telecom domain in which both agent and user act on a shared environment, which the paper calls dual control. The appendix example is a customer with no service because airplane mode is on and the SIM is unseated; the agent has to guide the user through toggling the mode and reseating the card, and success is an assertion that service status is connected. The paper's task counts are 115 retail, 50 airline and 114 telecom, each run four times. Its authors state that extending domains still depends heavily on human experts, that the user-tool approach has not been applied back to airline and retail, and that the user simulator itself makes errors: an audit found errors in 47% of airline conversations, 13% of them critical, against 16% and 6% in telecom.

The benchmark has been revised since. The project site's changelog, read November 5, 2026, records a February 2026 audit that fixed more than 50 airline and retail tasks, correcting expected actions, ambiguous instructions and impossible constraints, and March 2026 additions of voice and knowledge variants. The repository states that v2.0 runs are not input-identical to v1.0.1, that the natural-language assertion judge changed from GPT-4.1 to a GPT-5.4 mini model, and that 2.0 scores are therefore not comparable with the 1.x leaderboard. The public page shows only the top three per board, with no submission dates and no pass^k column: its headline metric is pass^1, the single-attempt rate the benchmark was designed to look past.

MCP-Universe, from Salesforce AI Research and submitted in August 2025 by Ziyang Luo and colleagues, runs 231 tasks across six domains, location navigation, repository management, financial analysis, 3D design, browser automation and web search, over 11 real Model Context Protocol servers. Tasks look like asking for a stock price between two dates, planning a route, or finding a flight price for a future date. Scoring is execution-based with no language-model judge: 84 evaluators, of which 32 check static hand-collected answers and 48 fetch real-time ground truth such as weather or a repository's issue count. The authors argue a model judge's frozen knowledge cannot grade real-time data and shows style bias.

The authors' stated limitations are candid. Execution-based evaluation needs far more human labour than a judge. One open-weight model could not follow the ReAct prompt and was run through a different agent SDK, which makes that comparison inconsistent. Claude Sonnet 4 was run without thinking mode. A commercial coding agent may have used its own internal tools rather than the benchmark's servers. And because the dynamic evaluators query live services, the ground truth moves over time. The leaderboard page shows no update date, and its overall track takes each model's better result from two harnesses, ReAct and native function calling, so the ranking mixes the two. Every model on it is from 2025.

MCPMark, submitted September 2025 by Zijian Wu and fifteen co-authors from the National University of Singapore's TRAIL group, LobeHub and EVAL SYS, has 127 tasks across five services: filesystem (30), Notion (28), Playwright (25), GitHub (23) and PostgreSQL (21). Each task starts from a curated initial state, requires create, read, update and delete operations, and is checked by a programmatic verification script. The examples are concrete: a browser login that must pass a Cloudflare Turnstile challenge, a GitHub continuous-integration setup with a linter, a database schema design. The paper reports an average of 16.2 turns and 17.4 tool calls per task and three metrics: pass@1, pass@4 and pass^4, where a task counts under pass^4 only if all four independent runs succeed.

The authors' limitations list includes the cost of the tasks, roughly three to five hours of expert effort each; difficulty that limits the benchmark's use for smaller models; a deliberately basic agent framework; low run-to-run consistency; and a note that local services are more common in training data and may explain part of the gap between local and remote services. What matters most for a reader is that the project now runs two leaderboards. The legacy board, stamped December 15, 2025, reports pass@1, pass@4 and pass^4 with a cost per run. The Verified board, stamped July 20, 2026, is described as a stabilised, version-pinned subset scored on a single run, with pass@4 and pass^4 marked not applicable and no cost column. The two are different series.

Put the two tables side by side and the top score goes from 57.5% to 96.1%. That is not model progress alone. It is a repaired and version-pinned task set, scored once instead of four times, with the strict pass^4 column removed. A launch post that quotes an MCPMark figure without saying which board it came from has left out the fact that decides whether the number is impressive.

Toolathlon, submitted October 2025 by Junlong Li and twenty co-authors including Graham Neubig and Junxian He, with HKUST's natural-language-processing group leading, spans 108 tasks across 32 applications and 604 tools, most exposed through MCP servers, with realistic starting states such as a course site populated with dozens of students and about twenty turns per task on average. The task index includes a Kubernetes security audit, merging datasets on a model hub, setting camera-ready deadlines for a conference, and building a personal website in Notion. Each task has its own execution-based evaluation script, and the leaderboard reports pass@1 with error bars, pass@3 and pass^3.

The original paper's headline was Claude Sonnet 4.5 at 38.6% success. Then, on June 30, 2026, the authors released Toolathlon-Verified, and their release note is the most useful single document on this page. The same 108 tasks, but artefacts changed across more than three quarters of the benchmark: 83 task packages with net changes, 76 with evaluator changes, 14 with changes to the instructions agents see. Their examples are specific. In one Notion task the instructions listed four sections while the evaluator checked for five, so, in their words, "An agent could follow the written task faithfully and still fail". In another, a comparison helper returned a tuple that was tested directly, so mismatches could slip past the failure path. A ground-truth value was corrected from 7.5 to 7. The note says errors ran in both directions, rejecting correct work and accepting incorrect work, and the leaderboard states that Verified begins a new score series not comparable with earlier versions.

Three of the five benchmarks publish a repeated-run metric beside the single-run one, and the gap between them is the most consistent finding across the boards. pass@1 is the share of tasks solved on one attempt, averaged. pass^k is the share of tasks solved on every one of k independent attempts. For a task that an agent completes 80% of the time, pass@1 counts 0.8 and pass^3 counts about 0.51, because three successes in a row at 80% is 0.8 cubed. The strict metric is the one that describes an agent you would leave unattended, which is why τ-bench introduced it and MCPMark's paper calls pass^4 a stricter measure of consistency.

The failure case is reading a mean as a guarantee. A vendor note that says scores were "averaged over 5 independent trials", as Anthropic's Opus 4.5 methodology note does, is reporting a mean pass rate, which is a pass@1-type number, not pass^5. The chart shows the same models on the same board under both metrics; the drop is between 8 and 20 points for every model where both are shown.

Vendors do document how they ran these benchmarks, usually in a footnote or a separate methodology page, and the documentation is where the comparability lives. Three examples, all from the vendors' own pages. Anthropic's Claude Sonnet 4.5 post of September 29, 2025 states in its τ²-bench footnote that scores used extended thinking with tool use and a prompt addendum to the airline and telecom agent policies, and a further addendum to the telecom user prompt to avoid the simulated user ending the interaction incorrectly. Anthropic's Opus 4.5 post of November 24, 2025 states a global method of a 64K thinking budget, default high effort and an average over five independent trials, and separately notes a τ²-bench airline case the benchmark scored as a failure because the model's way of helping was unanticipated.

Google DeepMind's Gemini 3 Pro evaluation methodology page, dated November 2025, states that its τ²-bench results use the standard Sierra framework with a prompt adjustment per environment, that the user model is Gemini with a system instruction, and that the headline is the average of retail, airline and telecom, which it gives as 85.3%, 73.0% and 98.0%. A single headline over a 25-point spread, with the vendor's own model playing the customer, is a legitimate number and a very specific one. The four questions below are the ones to ask of any such figure before it goes into a procurement document; the general method for turning them into your own test is in our guide to building an evaluation harness.

What was collected. For each of five tool-calling benchmarks, BFCL V4, τ-bench and τ²-bench, MCP-Universe, MCPMark and Toolathlon, with MCP-Bench noted: the builders, the task shape, the scoring rule, the size, the weaknesses the authors state in their own paper or release notes, and the top entries on each public leaderboard with the update stamp the page displayed. Vendor methodology statements were read from Anthropic's Sonnet 4.5 and Opus 4.5 posts and Google DeepMind's Gemini 3 Pro methodology page.

Sources. Primary pages only: the BFCL leaderboard page, its overall data file and the V4 web-search release note; the τ-bench and τ²-bench papers on arXiv, the project site and its repository; the MCP-Universe paper and leaderboard; the MCPMark paper and its legacy and Verified leaderboards; the Toolathlon paper, leaderboard and the Toolathlon-Verified release note; and the three vendor pages named above. Secondary coverage was not used for any figure. MCP-Bench is noted from its arXiv abstract only.

As-of date. All leaderboards and pages were read on November 5, 2026. Each table also carries the update stamp the page itself showed: BFCL April 12, 2026; MCPMark legacy December 15, 2025; MCPMark Verified July 20, 2026; MCP-Universe and the τ-bench site show no date. Scores on boards with an older stamp describe the models available at that stamp, not at the read date.

What we did not do. We ran no benchmark, no model and no harness. We did not obtain the full τ-bench leaderboard beyond the top three per board shown on its landing page, did not open the τ-voice, τ-knowledge or task-fixes papers, and did not read the primaries for Scale AI's MCP Atlas, Terminal-Bench or BrowseComp, which are therefore not covered. Where two documents from the same project disagree, as BFCL's scoring description does, both statements are reported and neither is chosen. This page is reviewed quarterly and when a benchmark publishes a repaired task set; each refresh updates the tables in place and moves the read date, and the URL stays stable.

The practical use of all this is a short test before any tool-use number is believed: which family, which harness, which metric, which series, and at what cost. If you want help turning that into an evaluation on your own tools, with your own policies and your own servers, it is part of what our AI transformation engagements cover; the framework choices are in our agent evaluation frameworks guide.