MultiLangBench

MultiLangBench is an AI coding benchmark introduced by OpenAI on October 19, 2026. It evaluates implementations of a SQLite-compatible database engine across 20 programming languages.^[1] The task gives a language model the SQLite 3.40.1 documentation and requires an executable implementation in a specified language.^[2]

Task and scoring

The reference-output script reads three SQL files for each test:^[3]

A test passes on an exact validation-output match. Crashes, mismatches, a 60-second timeout, and wrong-language or mixed-language implementations fail.^[2]

Scores pair API cost with test pass rate. Approximately 24,000 cases were converted from 51,000 SQLite Tcl tests. The recommended subset has 272 identifiers beginning 00, 01 or 02; full-suite runs should disclose that choice.^[2]

How the reference checks work

The repository's generate_goldens.py script asserts that the reference SQLite library is version 3.40.1. It creates input.db, copies it to output.db, then applies the user query with reference SQLite. It opens output.db read-only for validation. This constructs reference outcomes, rather than implementing the candidate.^[3]

Validation results become compact JSON arrays. The script represents binary values as objects containing a blob_hex field, retains Unicode characters during serialization, and uses the string Error for the SQLite corruption or SQL exceptions it catches during validation. Existing expected-output files are checked for disagreement, while missing files are written. These details matter when checking a runner against the supplied golden outputs.^[3]

Test identifiers also have an explicit construction. validate_hashes.py reads the three SQL files, serializes their named fields as compact UTF-8 JSON, and computes a SHA-256 digest. It raises an error if the digest differs from the directory name.^[4]

Evaluation environment

The candidate runs without internet access or installed SQLite implementations. Its executable accepts a database path as an argument and SQL through standard input.^[2]

The example cleanup script illustrates why checking only for a sqlite3 executable is insufficient. Its checks also cover shared libraries, Python bindings and tools with embedded SQLite. The script identifies Cargo as one such tool, and handles the modern Swift driver's SQLite dependency by configuring a legacy driver instead.^[5]

This script is deliberately disabled at its start: it prints a warning and exits before performing removals. Its later commands delete system binaries, development headers, libraries and package-cache files. It is environment-preparation code for an isolated evaluation container, not an installation step to run unchanged on a workstation.^[5]

Languages and reported results

The release covers C, C#, C++, COBOL, Common Lisp, Fortran, Go, Java, JavaScript, Lean, OCaml, PHP, PowerShell, Python, Ruby, Rust, Scala, Swift, Visual Basic and Zig.^[1]

OpenAI reported these average test pass rates at maximum reasoning effort:^[1]

The release estimates average API cost per sample at $3.48 for GPT-5.6 Sol and $0.77 for GPT-6 Luna. These are vendor-reported evaluation results and cost estimates, not independent measurements or API token prices.^[1]

What the score establishes

The shared interface is a database file.^[2] SQLite documents a structured on-disk format with a file header, fixed-size pages and B-tree structures.^[6] The evaluation therefore checks observable behavior through a file that reference SQLite can read.^[2][6]

A test pass rate should not be read as a production-readiness certification. SQLite's own testing program includes separate harnesses, fault injection, power-loss simulations, fuzzing, malformed-file tests and resource-leak checks.^[7] Passing a selected set of converted cases does not establish equivalent coverage of those checks.^[2][7]

MultiLangBench also differs from HumanEval, the evaluation introduced in the 2021 paper Evaluating Large Language Models Trained on Code. HumanEval measures functional correctness for programs generated from docstrings, with the paper studying Python code generation.^[8] MultiLangBench instead evaluates an executable database implementation using a common file-based interface.^[2] Their percentages therefore describe different tasks and scoring procedures.

Distinction from the COBOL migration evaluation

OpenAI's accompanying COBOL migration evaluation is separate. It converts existing COBOL programs to C++, Java, Kotlin, Python or Rust and requires byte-for-byte output parity on hidden tests; a migration succeeds only if every test passes. Its migration accuracy must not be substituted for MultiLangBench's test pass rate.^[1]

References

1. ^1 ^2 ^3 ^4 ^5OpenAI. "Advancing Coding in Low-Resource Programming Languages". October 19, 2026.
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8OpenAI. MultiLangBench README. Evaluation specification, inspected October 20, 2026.
3. ^1 ^2 ^3OpenAI. `scripts/generate_goldens.py`. Reference-output generation, inspected October 20, 2026.
4. ^OpenAI. `scripts/validate_hashes.py`. Test-identifier validation, inspected October 20, 2026.
5. ^1 ^2OpenAI. `scripts/remove_sqlite.sh`. Example environment cleanup script, inspected October 20, 2026.
6. ^1 ^2SQLite. "Database File Format". Inspected October 20, 2026.
7. ^1 ^2SQLite. "How SQLite Is Tested". Inspected October 20, 2026.
8. ^Chen, Mark, et al. "Evaluating Large Language Models Trained on Code". arXiv:2107.03374, July 2021.