How to Run a 1M-Row LLM Job Without Losing Money

A million-row LLM job is not a big version of a small one. Batch APIs change the cost per row by half, and a cheap model tier whose standard-list input rates sit under $0.30 per million tokens changed it again. What did not change is the failure surface: a crashed worker that resubmits, a result file that comes back out of order, and output that is not reproducible even at temperature zero.

Every one of those failure modes costs money rather than throwing an exception. A duplicate submission bills twice and produces two plausible answers. An out-of-order result file silently misaligns row 412,000 with row 411,000 if you map by position. A nondeterministic model turns a green regression suite into a coin flip. None of this shows up in a proof-of-concept over a thousand rows, which is exactly why it lands in production.

This guide covers the six things that are specific to bulk LLM work: which vendors actually publish a batch API and what each one caps, the submission and result contract, the idempotency ledger you have to build yourself because nobody ships one, partial-failure recovery, statistical QA for nondeterministic output with the sample-size math shown, and cost guards that hold when the platform limit is only approximate. Each vendor claim below is attributed to that vendor’s own documentation, with the primary sources linked at first mention; the QA sample-size table is computed from a formula printed on the page rather than sourced from anyone.

The first architectural decision is not which model to use. It is whether the model you want is reachable through a batch endpoint at all, because the 50% discount is the single largest cost lever available on a bulk job and it is not universal. Most published coverage of this topic is single-vendor: an OpenAI page about OpenAI’s batch API, an Anthropic doc about Anthropic’s. Put the four providers most bulk teams actually shortlist into one table and the gaps become the finding.

Two cells in that table are worth more than the rest. The first is Alibaba’s completion window. OpenAI and Anthropic both fix theirs at 24 hours; Model Studio lets the caller name a window anywhere from 24 to 336 hours. For a job that genuinely does not need same-day results — a backfill, a corpus re-classification, a one-off enrichment pass — a 14-day ceiling removes the expiry-risk problem that forces everyone else to shard aggressively. Its supported model list also includes DeepSeek R1 and V3, which means a batch layer for DeepSeek weights exists even though DeepSeek itself does not run one.

The second is the idempotency column, and it is the reason this post exists. Neither OpenAI nor Anthropic documents a request-level idempotency key on its batch product — a finding that only emerges from reading two vendor documentation sets against a third surface, OpenAI’s payments API, which does publish one. Model Studio’s batch reference was not audited for the same header in this pass, so treat that cell as unchecked rather than as a confirmed negative.

On price, label the surface every single time. Anthropic’s batch rate is 50% of standard across all supported models — Claude Opus 5 is $5.00 in / $25.00 out per million tokens on standard, and $2.50 / $12.50 on batch. OpenAI’s own pricing documentation shows the same halving on its batch and flex surfaces: GPT-5.6 Luna lists at $0.20 in / $1.20 out per million tokens on standard and $0.10 / $0.60 on batch. Aggregator listings frequently show the batch number in a column labelled as list price, and the mistake is invisible unless you check against the vendor’s own pricing page. A 2× error in the input rate of a ten-million-row job is not a rounding difference. Our breakdown of the August 2026 cheap-tier repricing walks the same discipline across the whole tier.

Every batch API that exists has converged on the same contract, which is unusual enough to be worth naming: JSONL in, a caller-supplied identifier on every line, a success file out, and a separate error file out. OpenAI requires custom_id, method, url and body on each input line and supports chat completions, responses, embeddings, moderations, image generation and video endpoints. Anthropic requires a custom_id matching ^[a-zA-Z0-9_-]{1,64}$. Alibaba’s interface is deliberately OpenAI-compatible and takes the same shape. If you build your submission layer around that contract, it ports.

Here is the assumption that quietly costs teams real money: custom_id looks like an idempotency key, so people treat it as one. It is not. In both vendors’ documentation its scope is result-matching inside a single already-submitted batch. Nothing in either doc states that resubmitting the same batch with the same identifiers — after a crash, a timeout, an ambiguous 500, or a deploy that replayed a queue — is detected, deduplicated or discounted. Submit the same 400,000 rows twice and you are billed for 800,000 rows and left holding two sets of plausible answers with no principled way to choose between them.

It is not that idempotency is an unsolved problem. It is that the LLM vendors have not implemented it at the model-call layer. OpenAI does publish a first-class Idempotency-Key header where the same key with the same parameters replays the cached result and the same key with different parameters returns a 409 conflict — but only in its Agentic Commerce API, the payments surface. Chat Completions, the Responses API and the Batch API have no equivalent. Anthropic publishes none at all. The OpenAI SDKs do attach an internal idempotency header on their own automatic retries for connection errors, 408, 409, 429 and 5xx responses, which is useful and entirely insufficient: it protects one call’s internal retry loop, not a worker process that died and came back.

The practical build is unglamorous. Give every row a stable, caller-generated identifier derived from the source record and the job version — not an autoincrement, because an autoincrement changes when the input is re-read. Write it to a ledger table before submission with a status column. Check the ledger for already-processed rows before building the JSONL, not after the results come back. Store the provider’s batch identifier against the row so a partially completed batch can be resumed rather than re-run. Then make the whole thing enforceable at the database level with a unique constraint on the row key, so a duplicate submission fails loudly at insert time instead of succeeding expensively at the API.

Read the direction of travel here as a market signal rather than an oversight. Batch APIs were built for throughput on jobs assumed to be re-runnable and cheap; the pricing and the semantics both reflect that. As bulk inference moves from experiments into scheduled production pipelines — nightly enrichment, continuous re-classification, standing moderation passes — the absence of a dedup primitive stops being a rough edge and starts being a billing hazard. Payments APIs solved this a decade ago and the IETF is standardising the header now. It would be reasonable to expect one of the three batch vendors to ship a documented key in the next cycle or two. Until one does, the ledger is not optional infrastructure.

A batch does not succeed or fail. Individual requests inside it reach one of several terminal states, and the recovery logic differs by state. Anthropic documents four — succeeded, errored, canceled and expired — and states explicitly that requests which error, are canceled, or expire are not billed. It also states per-row isolation as a design property rather than an implementation detail. OpenAI writes failures into its own error file with messages including the expiry case, phrased as a request that could not be executed before the completion window expired.

Anthropic’s own written guidance points the same way — monitor batch status regularly, implement retry logic for failed requests, use meaningful custom_id values, and consider breaking very large datasets into multiple batches for manageability. Sharding is the single highest-leverage decision here, and it is worth being deliberate about the size. Anthropic caps a batch at 100,000 requests or 256 MB; OpenAI at 50,000 requests or 200 MB per file with up to 2,000 batches created per hour. Those are ceilings, not targets. Sub-batches well below the cap give you finer-grained resume points, smaller blast radius on an expiry, and a per-shard cost checkpoint you can actually act on.

One retention detail catches people. Anthropic keeps batch requests and responses for up to 29 days measured from batch creation, not completion, after which the metadata remains visible but the results can no longer be downloaded. If your pipeline treats the provider as durable storage for a long-running reconciliation, that clock is shorter than it looks. Pull results into your own store as soon as a batch completes.

Set temperature to zero, use greedy sampling, send the identical prompt twice to a production endpoint, and you can still get two different answers. Most engineers who notice this reach for the standard explanation — floating-point non-associativity plus GPU concurrency — and stop there. Thinking Machines Lab published a piece in September 2025 showing that explanation is wrong, because GPUs do reproduce bitwise-identical matrix multiplication results across runs despite both of those conditions being present.

The actual cause they identify is that most GPU kernels — RMSNorm, matmul, attention — are not batch-invariant. The same input element produces slightly different numeric results depending on the size of the batch it happens to be processed alongside, and on a shared production endpoint the batch size is a function of how much other traffic arrived in the same millisecond. Your request’s numerics depend on other people’s requests. That is a structural property of production serving, not a temperature setting you forgot to turn down.

The team demonstrated the effect by sampling completions at temperature zero from a large open model on their own inference stack, finding a meaningful spread of distinct outputs that began diverging partway through generation, and then eliminated it by building batch-invariant kernels — at a real latency cost. Two caveats matter for anyone reading this as a benchmark. The specific divergence rate they measured belongs to that model on that stack and should not be treated as a constant that transfers to every vendor’s production API. And the fix is available to people who control their own serving layer, which is not you if you are calling a hosted batch endpoint.

The consequence for pipeline design is direct and non-negotiable: you cannot validate a bulk LLM job by diffing its output against a fixed golden file. Re-running the same prompts is not guaranteed to reproduce the same bytes, so a byte-level regression suite will throw false failures forever and everyone will start ignoring it. What you can validate is the distribution of quality across the output, which is a statistical question with a statistical answer.

Manufacturing solved this problem before software existed. Acceptance sampling takes a random sample from a production lot, measures the defect rate in the sample, and accepts or rejects the whole lot on that evidence rather than inspecting every unit. Substitute “completed sub-batch” for “lot” and it maps onto bulk LLM output cleanly: define an acceptable row-level error rate before the job runs, pull a properly sized random sample from each finished shard, and decide release-or-re-review on the sample’s measured defect rate with a confidence interval attached — not on someone eyeballing the first twenty rows.

The sample size comes from the standard formula for estimating a proportion, with a finite-population correction because your batch is a known finite size: n₀ = z² · p · (1 − p) / e², then n = n₀ / (1 + (n₀ − 1) / N). Using the conservative p = 0.5 assumption (which maximises the required sample when you have no prior estimate of your error rate), z = 1.96 for 95% confidence and z = 2.576 for 99%, and rounding every result up to a whole row, the table below is what falls out. Every cell is computed from that formula — no vendor publishes a recommended QA sample size, and you can reproduce all of these in a spreadsheet.

That flat line is the whole point. The instinct on a large job is to review a fixed percentage — “we’ll spot-check 1%” — which produces 10,000 rows of human review on a million-row batch and 10 rows on a thousand-row batch. Both numbers are wrong in opposite directions. The statistically required sample climbs from 278 to 385 across three orders of magnitude and then stops. At roughly two minutes of reviewer time per row, a million-row job needs a little under thirteen hours of structured review to make a defensible release decision, and a thousand-row job a little over nine. The two-minute figure is an illustrative planning assumption, not a measured constant — substitute your own and the shape of the conclusion does not change.

Two honest caveats on the method. First, this sizing estimates an overall defect proportion; if you need per-category confidence — error rates by document type, by language, by source system — you need a stratified sample, which means this n per stratum, not in total. Second, some practitioner writeups describe layering a statistical-process-control gate on top, flagging a batch when a monitored quality metric drifts outside roughly a mean plus or minus 1.5 standard deviations band across sampled outputs. That is one documented community pattern rather than an established standard, and it is worth treating as a starting hypothesis you calibrate against your own data. Where it does help is in reframing the question from per-row pass/fail to variance monitoring, which pairs naturally with tracing and cost observability for LLM pipelines.

Sampling QA is the second layer. The first is making malformed output structurally impossible. OpenAI’s Structured Outputs feature — response_format with strict: true, or strict function calling — constrains token sampling itself, so the model cannot emit a token that would violate your schema. That is a sampling-time guarantee rather than a validate-after-the-fact check, and it is available on the batch path too, because the Batch API’s supported endpoints include chat completions and responses, which carry the same parameter. Our complete guide to OpenAI Structured Outputs covers the schema constraints and the subset of JSON Schema it accepts.

On a million rows, that changes the economics of parsing. Without it, some fraction of rows come back with a stray preamble, a trailing comma, or a hallucinated extra field, and you build a repair stage that re-prompts the failures — a second pass that costs money and introduces its own nondeterminism. With it, the parse stage becomes deterministic and the repair stage disappears. For a bulk job, that is usually a larger practical win than the model choice.

A useful stress case is bulk synthetic-data generation, where both layers get exercised hard and a valid-but-wrong row propagates straight into a training set — the failure mode is covered in our decision guide for LLM synthetic-data generation. The general rule for any bulk job: schema-constrain everything you can express as a schema, and sample-verify the semantics you cannot. Getting that split right for a specific dataset is most of the work in the pipeline builds we run through our AI and digital transformation engagements, and it is the part that decides whether a bulk job is repeatable or a one-off.

The last layer is the one people set up on day one and never test. OpenAI lets an organisation or project set a monthly spend limit and toggle enforcement of a hard limit; once enforced, requests over the limit return HTTP 429 with insufficient_quota and traffic actually stops. A spend alert is a different mechanism entirely: it emails you and shows a dashboard banner while the key keeps working and the charges keep accruing. For an unattended overnight job, a soft alert is not a cost guard. It is a notification that you already spent the money.

Even the hard limit deserves less trust than its name implies at batch scale. Anthropic’s own documentation warns that because of high batch throughput, batches may go slightly over a Workspace’s configured spend limit. That is the vendor telling you its hard cap is approximate under exactly the conditions a bulk job creates. The design conclusion is not to abandon platform limits — keep them, hard-enforced — but to layer your own checkpointed counter underneath them.

The implementation follows directly from the sharding decision in section 04. If the job is submitted as sub-batches rather than one enormous file, you have a natural checkpoint between them: after each shard completes, add its actual token usage to a running total, and refuse to submit the next shard if the total exceeds the budget authorised for the job. That is a control you own, it is exact rather than approximate, and it degrades gracefully — a job that stops at 60% with 60% of the rows written and a resumable ledger is a recoverable situation. A job that runs to completion at four times the estimated cost is a conversation with finance. The same per-workload budgeting logic applies to agent runs, which we work through in our agent token-budget and cost-control framework.

One more lever, with an honest caveat attached. Prompt caching stacks with the batch discount, which matters enormously on bulk jobs where a long shared system prompt and few-shot block precede every short row-specific payload. But Anthropic is explicit that cache hits are best-effort inside a batch, because requests process concurrently and in no guaranteed order — it reports observed hit rates ranging from 30% to 98% depending on traffic patterns. Model your cost projections on the pessimistic end of that range and treat a high hit rate as upside, not as the plan.