SciCode

Scroll sideways for more →

SciCode is a research coding benchmark that asks large language models to write Python code for realistic, PhD level scientific problems drawn from working scientists' day to day workflows. It contains 80 main problems decomposed into 338 subproblems across 16 subfields in mathematics, physics, chemistry, biology, and materials science. SciCode was introduced by Minyang Tian and 29 collaborators in the paper "SciCode: A Research Coding Benchmark Curated by Scientists" (arXiv:2407.13168, July 2024) and accepted to the NeurIPS 2024 Datasets and Benchmarks Track.^[1][2][3]

The project is led from the University of Illinois Urbana-Champaign, Carnegie Mellon University, and Argonne National Laboratory, with contributions from MIT, Harvard, the University of Chicago, Stanford, and Princeton. Unlike exam style benchmarks such as HumanEval or MBPP, SciCode targets the kind of code that produces published results, including numerical methods, simulations, and quantitative modeling.^[1][4]

SciCode is notably difficult. The authors report that "Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting," meaning fully correct solutions to just 4.6% of the 65 main test problems without background notes.^[1] Progress since has diverged by metric. On the strict main problem metric, where every subproblem in a problem must pass, the best recorded score is still only about 9% to 11% (o3 mini and o4 mini variants, 2025), so the benchmark remains far from saturated.^[5][6] On the easier subproblem metric with scientist background supplied, 2026 frontier models such as Gemini 3.1 Pro, GPT-5.4, and Claude Fable 5 have climbed above 50%, reaching roughly 60% pass@1 on the Artificial Analysis leaderboard.^[8][9]

Background and motivation

Research software is messy in ways that classroom problems are not. A scientist usually has to combine knowledge of an underlying physical theory, a numerical method that handles stiffness or stability, an implementation in NumPy or SciPy, and a way to validate the output against a known limiting case. Existing code generation benchmarks largely ignore this layered structure: HumanEval evaluates short self contained functions, SWE-bench targets software engineering bug fixes, and competitive programming sets focus on puzzles.^[1]

The SciCode authors wanted a benchmark that reflects how scientists actually use code. Many problems were sourced directly from scripts the contributors had written for their own published research, then rewritten so that the solution path is well defined and a hidden numerical test exists for every subproblem. Several problems reproduce methods from Nobel Prize winning research: the self consistent field (SCF) method for density functional theory (Chemistry, 1998), the PMNS matrix for neutrino oscillation in matter (Physics, 2015), the Haldane model for the anomalous quantum Hall effect (Physics, 2016), optical tweezer simulations for microscopic thermodynamics (Physics, 2018), and the replica method for spin glasses (Physics, 2021).^[1][2]

Dataset composition

The full benchmark contains 80 main problems and 338 subproblems. The authors release a development split (15 main problems, 50 subproblems) for prompt engineering and a held out test split (65 main problems, 288 subproblems) used for the public leaderboard. The breakdown of main problems by subfield follows the table below.^[1][2]

Scroll sideways for more →

Main problems are split into between 2 and roughly 15 subproblems, ordered so that earlier steps can be reused as helper functions in later ones. Subproblems are written as Python function signatures with a docstring that describes the scientific task, inputs, and expected outputs. Test cases live in a numerical results file (test_data.h5) and many of them check agreement with closed form analytical solutions, published results, or independent reference implementations.^[1][2][6]

Problem structure

Every subproblem combines four kinds of difficulty: knowledge recall (retrieving the relevant scientific facts), mathematical reasoning (deriving or rearranging the right equations), algorithm design (picking and adapting a numerical method), and code synthesis (writing a runnable Python function). The model is also required to remain consistent across subproblems, since later steps typically import the solutions of earlier ones.^[1]

Scroll sideways for more →

Reproducing methods like density functional theory, BCS superconductivity calculations, or Diels Alder reaction modeling from scratch in Python requires both conceptual understanding and the bookkeeping discipline that real research demands.^[1][2]

How is SciCode scored?

SciCode runs in two main settings. In the standard setting, the model receives only the function signature, a brief docstring, and any imports already produced for earlier subproblems. In the with background setting, the model also gets a scientist authored note that explains the relevant physics, equations, or algorithm. The two settings let researchers separate two skills: how much a model knows about a field versus how well it can implement an already explained method.^[1]

Scroll sideways for more →

Generated code is executed against the hidden numerical tests with np.allclose style comparisons. A subproblem counts as correct when all of its tests pass; a main problem counts as correct only when every subproblem in it passes, which is why main problem accuracy is far lower than subproblem accuracy. The paper defines the bar precisely: "The LM is considered to have successfully solved the main problem when all subproblem solutions are correct and the integrated solution to the main problem is correct."^[1] The original harness used a two step pipeline (gencode.py then test_generated_code.py); newer evaluations integrate with the Inspect AI framework maintained by the UK AI Safety Institute.^[2][6]

Quality control involved three rounds of validation: an in domain scientist reviewing each problem and its tests, an out of domain scientist checking clarity, and a GPT-4 pass used to flag ambiguous prompts. Dependencies are kept to widely used libraries such as NumPy, SciPy, and SymPy.^[1][2]

Headline results

The original paper reported pass@1 scores for ten models. Claude 3.5 Sonnet led the standard setting with 4.6% main problem accuracy, with GPT-4o, GPT-4 Turbo, Gemini 1.5 Pro, and Claude 3 Opus clustered near 1.5%. Subproblem scores were much higher, since many subproblems are isolated helper functions that do not require all earlier code to be perfect.^[1]

Scroll sideways for more →

Adding scientist annotated background boosted performance, especially for reasoning models. Claude 3.5 Sonnet jumped to 35.4% on subproblems and 12.3% on main problems. OpenAI's o1 mini gained the most from background knowledge and topped that chart at around 13.8% pass@1 on main problems in some reports, which suggests knowledge gaps, not pure reasoning, are a major bottleneck.^[1][7]

What is the current best score on SciCode?

SciCode is reported two different ways, and the two numbers look very different. The main problem metric counts a problem correct only when every subproblem passes and the integrated solution runs; the subproblem metric scores each step independently, usually with scientist background supplied. Because a single main problem can contain up to 15 subproblems, main problem accuracy is far lower than subproblem accuracy.^[6][8]

On the main problem metric, the maintained leaderboard at HAL (Holistic Agent Leaderboard) hosted by Princeton tracks both raw accuracy and dollar cost per evaluation across newer models and agent scaffolds. As of its 2025 snapshot the top entries include o4 mini and o3 variants, GPT-4.1, and Claude Opus 4.1 running under either zero shot or tool calling SciCode agents, none of which clears 10%.^[5]

Scroll sideways for more →

The GitHub leaderboard also lists o3 mini variants between roughly 9% and 11%, records the single highest main problem score to date at 10.8% (o3 mini, low reasoning), and reports DeepSeek R1 at 4.6% main problem accuracy, matching Claude 3.5 Sonnet from the original paper. HAL has since paused adding new models to concentrate on measuring agent reliability, so its main problem figures reflect 2025 systems.^[5][6]

Subproblem scores tell a faster moving story. Artificial Analysis, which scores the 288 test subproblems pass@1 with scientist background included, reports that 2026 frontier models have pushed past 50%, roughly double the best 2024 figures. As of mid 2026 the leaders sit near 60%.^[8][9]

Scroll sideways for more →

Even at 60% subproblem accuracy, no system comes close to solving whole research problems reliably. The two metrics together capture the state of play: individual scientific coding steps are increasingly within reach, while stringing 10 to 15 of them into a correct, self consistent program is not.^[8][9]

Is SciCode saturated?

No. As of mid 2026 SciCode remains unsaturated on its headline main problem metric: the best recorded score is about 10.8% (o3 mini, low reasoning), and most frontier systems still sit below 10%.^[5][6] Subproblem accuracy with background has passed 50%, but solving a complete research problem end to end, where every subproblem must be correct and mutually consistent, is still out of reach. That is the gap the benchmark was built to expose, and it has persisted longer than for almost any other coding evaluation.

Common failure modes and example problems

Error analyses in the paper surface a recurring pattern: models often produce code that looks reasonable but uses a wrong sign convention, an outdated empirical formula, or an unstable algorithm. Cross subproblem consistency is another sticking point, where later functions expect shapes earlier ones do not return. Problems that combine two separate ideas, such as a finite difference scheme with a custom boundary condition, also push models past their reliability point.^[1][7]

Scroll sideways for more →

The public dataset spans a wide range of real research tasks. A few representative examples are listed below.^[1][2]

Scroll sideways for more →

How does SciCode compare to other coding benchmarks?

SciCode sits in a small but growing group of benchmarks for science focused code generation and reasoning, complementing more famous coding evaluations.

Scroll sideways for more →

SciCode's combination of executable tests, scientist authored gold solutions, and broad scientific scope makes it one of the few standardized ways to track progress on AI assistants for research computing.^[1][3]

Limitations

The paper and follow up commentary note several limitations. The benchmark is Python only, so it does not test Fortran, C++, or Julia. The dataset is fixed, which means scores can drift as public solutions seep into training corpora, though the held out test split helps. The 16 subfields are a sample of natural science computing, not a complete map; computational social science, neuroscience, and large scale climate simulation are out of scope. Evaluation also requires running scientific Python code with real dependencies, which adds engineering overhead.^[1][2]

Scroll sideways for more →

Reception and use

SciCode has become a standard reference in discussions of AI for science and the question of whether LLMs can do real research work. It is tracked by Artificial Analysis, the HAL leaderboard at Princeton, and Inspect AI evaluation suites, and frontier releases from Anthropic, OpenAI, Google DeepMind, and DeepSeek have included SciCode scores in technical reports or third party comparisons.^[5][8]

Where state of the art models now solve the majority of HumanEval problems, SciCode keeps showing single digit main problem accuracy years after release. That gap preserves a meaningful signal for future models to climb against, which is the job of a benchmark.^[1][5]

See also

• Code generation
• LLM evaluation
• HumanEval
• SWE-bench
• GPQA
• Inspect AI
• Reasoning models
• AI for science
• Scientific computing

References

1. Tian, M., Gao, L., Zhang, S. D., et al. "SciCode: A Research Coding Benchmark Curated by Scientists." arXiv:2407.13168, July 2024. https://arxiv.org/abs/2407.13168 ↩
2. SciCode official website. https://scicode-bench.github.io/ ↩
3. NeurIPS 2024 Datasets and Benchmarks Track poster page for SciCode. https://neurips.cc/virtual/2024/poster/97822 ↩
4. NeurIPS 2024 proceedings PDF, "SciCode: A Research Coding Benchmark Curated by Scientists." https://proceedings.neurips.cc/paper_files/paper/2024/file/36850592258c8c41cecdaa3dea5ff7de-Paper-Datasets_and_Benchmarks_Track.pdf ↩
5. HAL Princeton SciCode leaderboard. https://hal.cs.princeton.edu/scicode ↩
6. SciCode GitHub repository. https://github.com/scicode-bench/SciCode ↩
7. OpenReview discussion thread for SciCode. https://openreview.net/forum?id=ADLaALtdoG ↩
8. Artificial Analysis SciCode evaluations page. https://artificialanalysis.ai/evaluations/scicode ↩
9. llm-stats SciCode benchmark leaderboard. https://llm-stats.com/benchmarks/scicode ↩