Eight Headless Coding Agents, One Task: Tokens and Cost

This is a headless coding-agent cost-per-task benchmark: one small, fixed Python task run through eight agent CLIs — Claude Opus 5, GPT-5.6 Sol, Gemini 3.7 Flash, Grok 4.6, Kimi K3, Qwen3.8-Max, GLM-5.3 and DeepSeek V4 Flash — twice each, with the tokens each CLI reported, the seconds each took, whether the result actually passed its tests, and what those tokens cost at the vendor’s published rate.

It is an original measurement, not a survey. Nothing here was estimated: every token count was read from the CLI’s own output (or, for one CLI that prints none, from its local session log), and every pass/fail comes from us running python3 -m unittest -v in the agent’s directory rather than trusting the agent’s “DONE”. The complete 18-row dataset is in section 03; the prose around it is packaging.

Two things turned out to matter more than the headline cost figures. First, the harnesses themselves report cost inconsistently — one CLI’s dollar figure matches our rate-card calculation to the cent on its own vendor and is 27× wrong on a third-party endpoint, another’s is exactly 17% of its published rate card with no explanation found. Second, the cheapest-looking agent produced no usable output on the invocation most people would write. Both are in the table, not footnoted out of it.

The task was chosen to be small, unambiguous and self-verifying: a command-line CSV deduplicator plus a three-test unittest suite, standard library only, with an exact stdout contract. It fits in two files of roughly 140 lines and has a binary outcome — either python3 -m unittest -v passes three tests in the agent’s directory or it does not. The prompt, verbatim:

You are in an empty directory. Build a small Python 3 command-line tool and its tests, using ONLY the standard library.

1. Create dedupe.py. When run as `python3 dedupe.py contacts.csv deduped.csv` it must:
   - read the input CSV, which has a header row with exactly the columns name,email,phone
   - normalise email (strip surrounding whitespace, lowercase) and phone (keep digits only)
   - drop duplicate rows by normalised email, keeping the first occurrence
   - write the surviving rows with their normalised email and phone to the output CSV, with the same header
   - print one line to stdout: read=<rows read> written=<rows written> removed=<rows removed>
2. Create test_dedupe.py using unittest (not pytest) with three tests: (a) emails differing only by case or surrounding whitespace collapse to one row, (b) a phone like "+1 (555) 010-2020" becomes "15550102020", (c) an input file with only the header produces an output file with only the header and prints read=0 written=0 removed=0. Tests must create their own temporary input files.
3. Run `python3 -m unittest -v` and make all three tests pass.

Do not create any other files. When finished, reply with exactly one line: DONE

Each agent received the identical prompt as a single headless invocation, in its own freshly created empty directory, with its permission checks defeated so nothing would block on approval. All eight were launched at the same moment. Two full rounds were run (run1, run2), plus two additional Gemini-only re-runs that isolate a harness-configuration finding described in section 07. The eight invocations, exactly as executed:

Effort is therefore not equalised. Six agents ran at their explicit “high”; Kimi and Qwen ran at their CLI defaults, because neither exposes a comparable headless effort flag in the form used here. Kimi is always-thinking. That asymmetry is stated rather than hidden, and it is one reason this page does not rank models — our earlier reasoning-effort cost-vs-quality benchmarks show how much the effort dial alone moves token counts.

The table is the asset. Rows are ordered as the runs were recorded: the eight run1 rows, the two Gemini re-runs, then the eight run2 rows. “NOT REPORTED” means the CLI printed no such figure; it is not zero. “NO FILES IN CWD” means the agent replied DONE but nothing existed in its working directory for us to test. Token counts are as reported by each CLI; list price is calculated from them at the rates in section 04.

One chart, one column. List price per run is the single figure every reader can compare apples-to-apples across all eight agents: CLI-reported USD is NOT REPORTED for four of eight and wrong for two, and wall-clock is confounded by the concurrent launch. Run1 and run2 are shown side by side rather than averaged, so the run-to-run variance stays visible. The three Gemini runs that left no files in the working directory are drawn as dashed outlines rather than omitted: their cost was real even though the deliverable was not.

Reading across the rows rather than down the cost column: every agent that wrote files into its directory passed, on both runs. The fastest completion was Opus 5 at 20.2 s; the slowest was Qwen3.8-Max at 107.1 s. Opus 5 used almost no uncached input (4 and 6 tokens) but wrote 21K tokens to cache on each run, and at the $10 cache-write rate Table 2 uses, those writes were 76% and 71% of its two list-price figures. GLM-5.3 read the most from cache (203,392 tokens in run1). The two DeepSeek runs were within 2% of each other on price and within 3 s on wall-clock.

Seven of the eight rates below were read from the vendor’s own pricing or model page on the dataset’s as-of date; the eighth, Qwen3.8-Max, rests on a routing note plus secondary trackers and is marked as weaker in its own row. Where the page states a promotional or introductory period, the end date is recorded; where a rate depends on a threshold the runs did not cross, the tier that applied is named.

Three of these eight rates are explicitly promotional or time-bounded on the vendor’s own page — Sol’s promotional rate is guaranteed only through November 21, 2026, Gemini 3.7 Flash’s introductory rate runs through December 31, 2026, and Z.ai’s free cache storage is “for a limited time”. Anyone re-pricing these tokens later should expect at least the Gemini row to move. Our LLM API pricing index carries the broader rate card; the eight rows above are only the ones this dataset needs.

The first finding is about the instruments, not the models. Getting a comparable token split out of eight headless runs meant reading five different output schemas, and a sixth CLI offered none at all. The NOT REPORTED cells in Table 1 are the direct consequence.

The practical consequence for anyone building a cost dashboard on top of headless agents: there is no common field to read. Token splits have to be normalised per CLI, “input” means total input in some schemas and uncached input in others, thinking tokens are sometimes a separate count and sometimes folded into output, and a cost figure — where one exists — is not necessarily the vendor’s. Our per-task, per-user agent cost framework assumes you can get a clean per-task token count; this table is what obtaining one actually looked like.

Two CLIs print a dollar figure. Neither should be read without knowing what produced it. The comparison that makes this visible is already in Table 1: the CLI-reported USD column next to the list-price column, row by row.

Claude Code through a third-party endpoint. Pointed at Z.ai or DeepSeek via the Anthropic-compatible base URL, Claude Code still prints a total_cost_usd — computed with Anthropic’s price table, and it logs unrecognized_model while doing so. This is consistent with how Anthropic documents the field. Claude Code’s headless-mode documentation says of total_cost_usd and the per-model cost breakdown that “Both figures are client-side estimates and can differ from your actual bill.” The figure was never claimed to be a bill; on Anthropic’s own models it happens to be exact, and on someone else’s models it is Anthropic’s rate applied to someone else’s tokens. A closed issue on the Claude Code repository (#53371, filed April 25, 2026) separately reports the field returning roughly 10× too high for a same-vendor call — a different failure mode from ours, but corroboration that the field is documented as an estimate rather than a ledger. Anyone reading /cost against a third-party endpoint is reading a number that can be off by an order of magnitude. The column stays in Table 1 for exactly that reason.

Grok’s field is a different kind of number. xAI’s cost-tracking documentation describes the underlying cost field as “the actual amount billed, after all applicable discounts (including prompt caching reductions) have been applied, and inclusive of all token costs and server-side tool invocation costs” — a server-computed billed figure, not a client-side estimate. That is what makes the 0.17 ratio worth reporting rather than dismissing. A promotional or subscription-linked discount, an undisclosed caching credit, or a rate-card page that lags the billing system would each produce a gap like this; none of them is documented on any page we read, and the consistency across two independent runs rules out random variance. The page states the discrepancy with its exact numbers and does not pick a cause.

Cache reads are most of the input in all but one row. Across the 18 rows, uncached input ran from 4 tokens (Opus 5, run1) to 61,461 (DeepSeek, run2), while cache reads ran from 22,016 (Sol, run1) to 203,392 (GLM-5.3, run1). Expressed as a share of all input tokens (uncached + cache read + cache write), cache reads were between 46.0% and 95.4% in every run, and above half in seventeen of the eighteen — the exception is GPT-5.6 Sol’s run1 at 46.0%. The bars below show run1 per agent; run2 is in the sub-label.

The pricing consequence follows directly. Re-pricing each row with every input token at the uncached rate — the mistake a naive “input tokens × input price” estimate makes — would have overstated the full-run list price by between 1.4× (Opus 5, run1) and 4.8× (both Kimi runs) across these 18 rows. Table 1 prices cache reads at the vendor’s published cache-read rate wherever one exists, which is all eight vendors here.

The cheapest-looking agent and the working directory. Gemini 3.7 Flash’s list-price rows are among the lowest on the page: $0.0715, $0.0547, $0.0649 and $0.0349. Three of those four runs produced nothing in the directory the agent was launched in. Run1, with the cwd not in the CLI’s trusted-workspace list, errored after 60 s and 202K tokens with a message that artifacts must live under ~/.gemini/antigravity-cli/brain/<conversation>/ — status ERROR, final response still “DONE”. The first retry, with the cwd pre-trusted, returned SUCCESS and “DONE” with zero files in the cwd: the files had been written to ~/.gemini/antigravity-cli/scratch/, where they did pass the tests. Only the second retry, with --add-dir pointing at the cwd, put the files where the task said and passed in 71.1 s. Run2, launched without --add-dir, reproduced the global-scratch behaviour.

For a cost benchmark the consequence is that the cheapest-looking agent produced nothing usable, twice, on the invocation most people would write. Those rows stay in the table and in Figure 1, drawn as outlines: the tokens were consumed and would have been billed on a metered plan whether or not the deliverable arrived. A reader who wants a single Gemini figure for this task should take the --add-dir run, $0.0649, and note that it took the third attempt to obtain.

Table 1 carries two cost columns on purpose. List price is what the same tokens would cost on the vendor’s metered API. The plan column is what was actually paid, and for seven of eight agents the answer was nothing at the margin.

Summed across all 18 rows, the list-price column comes to $1.92 for 2,579,601 tokens; the amount actually paid was about $0.03. That gap is the reason the page refuses to call any agent “cheap” or “expensive” without naming the plan. A team on flat subscriptions experiences cost as quota pressure and rate limits; a team on metered keys experiences it as a bill; the same token counts sit under both. The off-peak detail matters too — DeepSeek’s rate doubles inside its published peak bands, so the same two runs inside a peak window would have cost about $0.07 rather than $0.03. Our earlier post on DeepSeek and GLM off-peak pricing windows defines the windows; this dataset is a dated data point inside one.

The limitations in the Methodology block are not boilerplate; each one rules out a conclusion a reader might be tempted to draw. One task, two runs, one time of day, one machine, concurrent launch. Wall-clock in particular is confounded: eight agents hitting eight vendors at the same second means each wall-clock figure includes whatever queueing that vendor applied at that moment, so a 20 s versus 107 s gap is a fact about this run, not a latency ranking. Effort was not equalised, and our effort benchmarks show that dial alone can move token counts by multiples.

What the dataset does support is narrower and, we think, more useful than a ranking. It shows the shape of a small headless coding task in 2026: a few thousand output tokens, uncached input running from single digits to tens of thousands, and cache reads that outweigh fresh input in seventeen of the eighteen rows. Because a cache read is billed at a fraction of the uncached rate, cache reads dominate the token count by more than they dominate the bill — which is why re-pricing them as fresh input inflates these rows by 1.4× to 4.8× rather than by an order of magnitude, and why that re-pricing is the estimating mistake worth catching first. It shows that the cost fields CLIs print are not interchangeable with each other or with a bill. And it shows that a DONE reply is not a deliverable: the one agent whose output we could not test said DONE every time.

Projecting forward, the part of this table most likely to change is not the token counts but the rate card under them. Two of eight rates are explicitly promotional with stated end dates and a third is free “for a limited time” with no end date given, one rests on a routing note rather than a pricing-table row, and one vendor doubles its rate by the clock. A refresh of this dataset will re-price the same tokens against whatever the pages say on that day, and the difference between the two list-price columns will be a measurement of rate-card drift rather than of the models. The harness findings are likelier to age the other way: a CLI that prints no usage, or prices third-party tokens with its own vendor’s table, tends to get fixed once enough people notice.

Two companion pieces frame where this sits. We defined the metric — cost per successful task, not cost per token — in our cost-per-successful-task post, with worked hypothetical comparisons; this page is the first real measurement behind it: on this task seven agents succeeded on every attempt, and the eighth needed a third attempt to deliver into the working directory. And every CLI here ran with its permission checks defeated — the census of coding-agent permission defaults records what each of these CLIs does when you do not. Teams deciding which of these agents to route which work to are the usual starting point of our AI transformation engagements, which begin with a measurement like this one on the team’s own tasks rather than ours.