AI COBOL Migration: What OpenAI's New Coding Results Actually Show

On October 19, 2026 OpenAI published results for its models on what it calls low-resource programming languages: COBOL, Fortran, Visual Basic and others with little code or documentation in public training data. The headline numbers are an 87.4% average test pass rate for GPT-6 Astra on a build-SQLite-from-scratch benchmark across 20 languages, and 97.9% for GPT-6.1 Sol on a COBOL-to-modern-language migration test at higher reasoning effort. Both evaluations are OpenAI's own, and both were run by OpenAI.

The reason this matters to a business reader is the installed base. OpenAI's post, citing IBM, says COBOL sits under more than 40% of online banking systems, 80% of in-person credit card transactions and 95% of ATM transactions. Fortran still runs weather and nuclear safety codes. The people who wrote these programs are often gone, the documentation is partial, and the business rules live only in the code. That is exactly the kind of estate where a migration either preserves behaviour exactly or quietly produces different financial results.

So the question for a CIO or engineering lead is not whether the scores are impressive. They are. The question is what a score on an OpenAI-built test tells you about your own code, and how to find out without betting a system on it. We read the post and the public benchmark repository on October 21, 2026 and set out what was measured, what was not, and how to run a pilot.

The post compares current models against GPT-5.6 Sol, OpenAI's own earlier model, on two evaluations it built for the purpose. That framing matters. Every improvement is a gap between two OpenAI models on an OpenAI test, not a comparison with a human team, a migration vendor or another lab's model. Read that way, the numbers are clean and worth having.

The four figures below are the ones OpenAI leads with. Each is stated as reported. The cost figure for GPT-6 Luna is an estimate of API spend per sample; the cost figure for the migration test is stated as a share of what GPT-5.6 Sol spent at the same setting. We explain the two cost claims in section 05, because the post gives the migration saving two ways.

One example of why the baseline matters: a 5.4% to 86.6% jump on migration reads like a sixteen-fold improvement, and arithmetically it is. But a 5.4% baseline also says the earlier model was close to useless on this specific test, so most of the gain is the test becoming passable at all. The failure case is quoting the ratio in a board paper as a measure of how much better migration has become in general. It is a measure of how much better one model got on one test.

MultiLangBench is the public half of the work. The task is to implement a SQLite-compatible database engine in a requested language, over an extended coding session, with the model expected to organise a codebase, manage state, handle errors and add features without breaking earlier ones. The repository on GitHub, released under Apache 2.0, states that the model is given the SQLite documentation and no internet access, and that every trace of SQLite itself is removed from the container. A solution written in the wrong language, or in a mix of languages, scores zero.

Scoring uses database files as the common interface. The candidate engine receives a starting database and a SQL query and must produce a modified database. The result is checked against what real SQLite 3.40.1 produces. OpenAI adapted the tests from SQLite's own suite; the repository says about 24,000 of roughly 51,000 TCL tests could be converted, and recommends scoring on a fixed subset of 272 tests with a 60-second limit per test. A crash, a mismatch or a timeout each count as a failure.

The per-language results are where the post is most useful, because they show which languages remain hard. OpenAI reports GPT-6 Astra above 83% in 19 of 20 languages, with COBOL the outlier.

Why COBOL lags is not stated in the post, but the task hints at it. Building a database engine means managing memory, strings and recursion in a language designed for fixed-format business records, with little idiomatic open-source code to learn from. The example to hold onto is Fortran: it has the same scarcity problem and reached 83.5%, so scarcity alone does not explain the gap. The failure case for a reader is treating 72.1% as a verdict on COBOL migration. It is a score for writing a new COBOL program of unusual shape, which is not the job most estates need done.

The second evaluation is the one a legacy estate cares about. OpenAI asked its models to migrate COBOL programs to one of five targets: C++, Java, Kotlin, Python or Rust. Success is judged on hidden tests that require byte-for-byte parity on observable outputs, and a migration counts as successful only if every test passes. That is a strict rule, and it is the right one for financial code, where a rewrite that compiles and runs can still be wrong.

The results, as OpenAI reports them, are in the table. The post does not say how many programs were migrated, how long they were, or how they were selected. It does not publish the test. Those gaps are what keep this a vendor result rather than a reproducible one.

The example worth copying is the pass rule, not the score. If your own pilot accepts a migration when most tests pass, you will get a higher number than OpenAI's and a worse system. The failure case is the quiet one: a migrated batch job that runs to completion and posts figures a few cents different from the original, discovered at month end. All-tests-pass is the only standard that catches it.

OpenAI's post gives one example of what a migration has to get right, and it is a good one. COBOL's fixed-point arithmetic throws away extra decimal digits unless the program asks for rounding. A rewrite in a language whose numeric types round by default will compile, run and produce different totals. Nothing crashes. The books are simply wrong.

That is one of many such rules. Missing-record handling, date formats, field padding, sign conventions and the exact order in which a batch processes its input all encode business decisions that nobody wrote down. A model that understands COBOL as a language is not enough; it has to reproduce the program as it behaves, which is why OpenAI's test checks outputs rather than code.

The practical consequence is that test coverage, not model choice, decides whether a migration can be trusted. OpenAI could score 97.9% because it had hidden tests that pin every observable output. Most estates do not. Building that harness is the first job of a pilot and usually the longest, and it is the part no model does for you. Our order-of-operations guide for AI transformation puts measurement before automation for the same reason.

OpenAI gives cost in two forms and they are easy to confuse. For MultiLangBench it reports an estimated API cost per sample: $0.77 for GPT-6 Luna against $3.48 for GPT-5.6 Sol, which it rounds to approximately 78% lower. That is a per-attempt figure for building an engine, and Luna reached only 40.3% average pass rate at that price, so it is a cheaper partial result, not a cheaper equivalent one.

For the migration test the post says GPT-6.1 Sol at low effort cost 39% less than GPT-5.6 Sol in its summary and, in the detailed section, that it ran at just 61% of the API cost. Those are the same claim stated two ways. No absolute dollar figure is given for the migration test, and nothing is stated for the higher-effort run that reached 97.9%, which will have cost more per attempt.

The example to run in your own spreadsheet: if a program takes two attempts at low effort for every one at higher effort, the cheaper setting may cost more in total once review time is counted. The failure case is choosing the setting by price per call and ignoring the price of a failed migration that a human then has to debug. Current list prices for these models are in our GPT-6.1 Sol pricing and upgrade guide and our GPT-6 Astra price and benchmarks guide.

Four things are missing from the post, and each one changes how the numbers should be used. First, there is no comparison with any other lab's model. Anthropic, Google and the open-weight labs have not published results on these tests, and OpenAI did not run them, so "best model for COBOL" is not a claim the post supports. Second, the migration evaluation is private: its programs, sizes and selection method are unpublished, so nobody outside OpenAI can reproduce the 97.9%.

Third, the sample is unknown. A test built from short, self-contained COBOL programs with clean inputs will score differently from one built on a 300,000-line batch system with CICS calls and copybooks shared across a dozen jobs. The post does not say which it used. Fourth, the task ends at parity. Nothing in either evaluation measures whether the migrated code is maintainable, idiomatic, or faster, and those are the reasons most organisations migrate in the first place.

OpenAI's own sentence on this is the one to keep. The post says that applying these capabilities to production software "still requires expert review and testing against the system's business rules and dependencies." That is not a disclaimer; it is the project plan. The example is a bank that migrates a rate-calculation module at 97.9% parity and discovers the remaining 2.1% is the leap-year handling. The failure case is a programme that budgets for the 97.9 and not the 2.1.

The only migration score that transfers is one measured on your programs with your tests. OpenAI's method is a good template: pick programs, capture their observable outputs on real inputs, migrate, and accept only when every output matches byte for byte. The sequence below is how we would scope four to six weeks, with the model as the last decision rather than the first.

One example of a result that changes the decision: if three of your ten programs fail on the harness itself because production inputs are not reproducible, the finding is that the estate is not ready to migrate by any method, and that is worth knowing before a vendor quotes. The failure case is the opposite: a pilot on three hand-picked programs that all pass, followed by a contract for the other 1,200. Our AI transformation practice runs the harness-first version of this with clients, and the broader model context sits in our GPT-6 Sol and Luna launch guide.