OpenAI's new evals put GPT-6 Astra at 87% on a SQLite build in 20 languages, COBOL included

OpenAI published two new coding evaluations on 19 October aimed at what it calls low-resource programming languages: COBOL, Fortran, Visual Basic and OCaml, languages with far less public code and documentation to learn from than Python or JavaScript. The post, Advancing coding in low-resource programming languages, comes with a public benchmark repository, openai/MultiLangBench, released under Apache-2.0.

Test one: build SQLite from scratch, in 20 languages

MultiLangBench asks a model to build a SQLite-compatible database engine in a given language over an extended coding session. To grade every language the same way, OpenAI uses database files as the interface: the candidate program gets an initial database and a SQL query and writes out a modified database, which the reference SQLite implementation then checks. The tests are adapted from SQLite’s own test suite.

OpenAI’s reported results, at maximum reasoning effort:

• GPT-6 Astra: 87.4% average test pass rate across 20 languages, up from 26.8% for GPT-5.6 Sol, and above 83% in 19 of the 20.
• COBOL stays the hardest at 72.1% (from 10.9%). Fortran goes from 21.4% to 83.5%, Visual Basic from 27.5% to 89.4%, OCaml from 25.6% to 85.8%.
• GPT-6 Luna: 40.3% average, at an estimated $0.77 per sample against $3.48 for GPT-5.6 Sol, which OpenAI rounds to about 78% cheaper.

Test two: migrate COBOL without changing behavior

The second eval asks models to port COBOL programs to C++, Java, Kotlin, Python or Rust. Hidden tests require byte-for-byte parity on observable output, and a migration only counts if every test passes. OpenAI’s example of why that matters: COBOL fixed-point arithmetic drops extra decimal digits unless rounding is requested, so a rewrite that rounds can run cleanly and still produce different financial results.

OpenAI says GPT-6.1 Sol scores 86.6% at low reasoning effort, against 5.4% for GPT-5.6 Sol at the same setting, at 61% of the API cost. At higher effort it reaches 97.9%, which OpenAI calls close to saturating the eval.

What to keep in mind

These are OpenAI’s evaluations, run by OpenAI, on OpenAI models, and the post compares only against older OpenAI models. The useful part for everyone else is that MultiLangBench is public, so you can read what it tests and run it against the model you actually use.

OpenAI also adds its own caveat: production migrations “still” require expert review and testing against the system’s business rules and dependencies. For a bank’s COBOL estate, a 97.9% pass rate on a benchmark is a reason to pilot, not a reason to skip the parity tests.