CritPt

CritPt (Complex Research using Integrated Thinking - Physics Test, pronounced "critical point") is a benchmark that tests large language models on unpublished, research-level physics problems. It was introduced in September 2025 by a team led by researchers at Argonne National Laboratory and the University of Illinois Urbana-Champaign, and it consists of 71 composite research challenges that are also broken into 190 smaller checkpoint tasks.^[1][2] Every problem was written by working physicists from their own research and has a machine-verifiable final answer, graded automatically against private reference answers.^[2] Artificial Analysis began running the benchmark in November 2025 and added it to its Intelligence Index in January 2026. A revised dataset, CritPt v2, was released on 22 October 2026, and a 40-challenge hard subset of v2 replaced the original benchmark in version 5.0 of the index.^[3][4][9]

When the paper was first posted, the best model without tools, GPT-5 (high), averaged 4.0% on the full challenges; later revisions of the paper report 5.7%, rising to about 10 to 12.6% with code and web tools.^[1][18][2] By late October 2026, the top-scoring models with a code-execution tool, GPT-6 Astra and GPT-6.1 Sol, scored above 80% on the full CritPt v2 set.^[8] Two separate audits published in 2026 reported that a substantial share of the original challenges were underspecified or ambiguous, and Epoch AI rated the original version "Flawed" in an October 2026 review.^[12][13][14]

Overview

Sources: ^[1][2][3][4]

Development and authors

The paper "Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark" was first submitted to arXiv on 30 September 2025 and revised on 1 October 2025, 20 November 2025 and 8 May 2026.^[1] The current version lists 65 authors. Minhui Zhu (Argonne National Laboratory) and Minyang Tian (Argonne and the University of Illinois Urbana-Champaign) are the equal-contribution lead authors and corresponding authors. The final authors listed are Nicolas Chia, Eliu Huerta and Hao Peng, with affiliations at Argonne, the University of Illinois Urbana-Champaign and the University of Chicago.^[1][2] Other contributors came from institutions including Caltech, Harvard, Columbia, Princeton, the University of Chicago, ETH Zurich, the Perimeter Institute and the University of Waterloo, plus several independent researchers.^[2]

According to the paper, the problems were written by more than 50 physics researchers at 30 institutions, including senior PhD students, postdocs and professors, each drawing on their own research. Every data contributor, coordinator, reviewer and science writer either held a physics PhD or was an active physics PhD student.^[2] The work was funded by Laboratory Directed Research and Development funding from Argonne, with computing time from the Argonne Leadership Computing Facility and the NSF-supported Delta and DeltaAI systems at Illinois.^[2]

Several of the core authors had previously built SciCode, a research coding benchmark curated by scientists, whose author list also includes Minyang Tian (first author), Minhui Zhu, Ofir Press, Eliu Huerta and Hao Peng. The CritPt paper says its answer-format criteria were partly inspired by SciCode, FrontierMath and TPBench.^[2][16]

The evaluation code was posted on GitHub and the 70 test challenge statements were released on GitHub and Hugging Face on 21 November 2025, the same day CritPt debuted on Artificial Analysis. The Hugging Face dataset card lists an Apache 2.0 license.^[3][5][6] Minhui Zhu presented the benchmark at the American Physical Society's Global Physics Summit on 17 March 2026.^[11]

Design

Coverage

The paper groups the challenges by research area using a modified version of the American Physical Society's Physics Subject Headings (PhySH) scheme. Because 33 challenges (46.5%) and 88 checkpoints (46.3%) span two or more areas, the area counts add up to more than the totals.^[2]

Source: CritPt paper, Table 1.^[2]

The challenges also cover theoretical, experimental and computational work. Experimental challenges cannot ask a model to run equipment, so they focus on designing or interpreting experiments under realistic constraints.^[2] Test challenge titles listed in the paper include "Holographic Weyl anomaly", "Zero temperature entropy in Sachdev-Ye-Kitaev models", "Magic wavelengths for Yb isotopes" and "LIGO with modified mirrors for ultralight vector dark matter".^[2]

Problem criteria

The authors set three main criteria for each problem:^[2]

• Search-proof but solvable. Problems are newly created so that their answers cannot be found with a web search, but they can be solved with public knowledge. Most fall into three types: modified versions of published results, non-trivial applications of a known method to a specific system, and intermediate calculation steps that papers leave out.
• Open-ended with verifiable answers. Questions are open-ended, with conventions and units stated explicitly. When an expression is too complex for reliable symbolic checking, the model must return a Python function, which is then run against test cases. Where a yes/no answer is unavoidable, several related questions are asked and all must be correct.
• Guess-resistant. Physics answers often take common values such as 0, 1/2 or pi, so each final answer usually includes at least one non-universal quantity, such as a floating-point number given to several decimal places, a large integer or a dimension-dependent symbolic expression.

Each challenge went through iterative revision between the problem author and coordinators (typically three or more rounds and up to ten), peer review by researchers in related areas, checks of the derivations by additional experts, and editing by a science writer. The paper says creating one full challenge took more than 40 hours of expert effort on average.^[2]

Challenges and checkpoints

Each challenge is a self-contained, research-style problem pitched at a junior researcher. A setup section supplies the background, much as a mentor would scope a project, and the challenge question then poses the central research task. The paper's physics contributors judged the challenges comparable to the "warm-up research exercises" a hands-on principal investigator might give junior graduate students.^[2] Each challenge is also split into two to four checkpoint questions that isolate specific steps, such as filling in part of a long derivation, solving a simplified version (for example, one dimension before higher dimensions) or analyzing a limiting case.^[2]

One challenge, "Quantum error detection", is public along with its checkpoints, solutions and error analysis. The solutions to the other 70 challenges are kept private to limit data contamination.^[2]

Evaluation method

Two-step answer generation

The original protocol separates reasoning from formatting. In the first step, the model writes a complete free-form solution. In the second step, it is asked to place its final answer in a provided Python code-block template. The authors say this avoids early conversions (for example into SymPy) that could distort the reasoning, and reduces parsing errors.^[2]

Automated grading

The grader extracts the answer function from the code block. Only a limited set of libraries (such as math, numpy, sympy and scipy) is allowed, and unsafe operations are filtered out. Answers are graded by type:^[2]

Code runs in a sandbox with a default 30-second wall-clock limit and bounded memory, or a higher limit set by the problem author for computationally heavy tasks.^[2]

Public grading server

Because the answers are private, the authors run an online grading server. It accepts only complete batches covering all 70 test challenges and allows 10 submissions per account in any 24-hour period. The server returns accuracy as the fraction of challenges solved.^[2][3] Artificial Analysis supports the grading server and provides free API access to it, granted case by case to approved labs and researchers.^[2][7]

Metrics

The paper reports average accuracy over five independent runs as its main metric. It also defines a stricter "consistently solved" rate, which counts a problem only if at least four of five runs are correct. Checkpoints are evaluated in sequence within one conversation in two settings: "self-carryover", where the model relies on its own earlier answers, and "oracle carryover", where it receives the expert answer to the previous checkpoint before attempting the next.^[2]

Results in the original paper

The paper evaluated 10 models in 2025. Reasoning models included GPT-5 (high), o3 (high), o4-mini (high), Gemini 2.5 Pro, Gemini 2.5 Flash, DeepSeek-R1 and Claude Opus 4. General chat models were GPT-5 (minimal reasoning), Llama 4 Maverick and GPT-4o. Only GPT-5 (high) was also tested with tools.^[2]

Source: CritPt paper v4, Table 3. Challenge figures are averages over five runs on the 70 test challenges; checkpoint figures cover 187 test checkpoints.^[2]

The authors drew several conclusions. Adding a code interpreter raised GPT-5 (high) from 5.7% to 10.6%, while adding web search on top gave only a small further gain to 12.6%, which they read as evidence that the search-proof design worked.^[2] All three general chat models scored zero on full challenges. Under the consistently solved metric, only the GPT-5 (high) configurations solved any challenge reliably; the base model did so for 3 of 70.^[2] The paper also noted that GPT-5, released near the end of data collection, showed only limited progress over o3 on full challenges, and suggested that a realistic benchmark of this kind could resist new model generations "for a substantial amount of time".^[2] The first arXiv version gave the best base-model figure as 4.0%, and that number also appears in the APS conference abstract.^[18][11]

Artificial Analysis leaderboard

The paper says Artificial Analysis independently validated its challenge results using the public pipeline, and its acknowledgments thank the company for hosting future evaluations on the 70 test challenges.^[2] The CritPt repository marks 21 November 2025 as the benchmark's debut on Artificial Analysis. Before the v2 update, the repository's leaderboard listed Gemini 3 Pro at 9.1% without tools, a figure it attributed to Artificial Analysis.^[3][4]

For the original benchmark, Artificial Analysis runs all 70 test challenges with the two-step prompting approach, five repeats each, and no agent harness or tools, and scores them on the official grading server.^[9][7] CritPt joined the Artificial Analysis Intelligence Index in version 4.0 (January 2026) and remained a component through version 4.3.2, with a 10% weight from version 4.2 onward.^[9] The original leaderboard is now labelled legacy. As read on 30 October 2026, it showed Gemini 4 Carbon (high) leading at 36.0%, followed by Muse Wonder Full (max) at 32.9% and GPT-5.6 Sol (max) at 32.3%. Several other frontier configurations scored between about 28% and 32%.^[7]

CritPt v2

On 22 October 2026, the benchmark authors pushed a commit titled "CritPt v2" to the GitHub repository. It revised the public challenge files and updated the README to say that CritPt "is an actively maintained benchmark" whose current deployed version is v2.^[4][3] A month earlier, on 26 September 2026, the project's news page had said: "We are actively maintaining the benchmark and incorporating community feedback. All benchmark problems are currently under expert review."^[6] Artificial Analysis describes v2 as the authors' revision of the 70 test challenges, reviewed with physics experts to correct "underspecified statements, errors, and typos". Epoch AI's review notes that v2 revised 47 challenges.^[8][14]

Artificial Analysis also changed how it runs the benchmark. For v2, it runs all 70 challenges in Stirrup, its open-source agentic harness, inside a Linux sandbox with no internet access and with Python 3.13, numpy, scipy, sympy and matplotlib installed. The system prompt is the CritPt authors' one-step prompt for models with code tools, and the model has at most 50 turns. It submits by calling a finish tool with its reason for finishing, its derivation and its answer in the problem's code template. The authors' grading server scores each repeat as one batch. A challenge scores zero if no answer is submitted, for example when the model runs out of turns, a provider safety filter blocks the request, or reasoning uses up the maximum output. Accuracy is averaged over five repeats.^[9]

Hard subset in Intelligence Index v5

Artificial Analysis announced Intelligence Index v5 in late October 2026. It replaced the original CritPt with "CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool", keeping the 10% weight. The full 70-challenge v2 is reported as a separate, standalone evaluation.^[10][9] According to the methodology, the hard subset is "a fixed, private list of 40 of the 70 CritPt problems, selected to remove the 30 problems that seven frontier models solved in all five attempts at the time CritPt v2 was launched". All 70 problems are run and graded, and the index uses mean accuracy on the 40 hard problems.^[9] Along with Terminal-Bench-Science, the hard subset makes up the index's Science category. The methodology marks it as using tools and as private.^[9]

Scores

The table below gives the configurations Artificial Analysis showed by default on its CritPt v2 page, as read on 30 October 2026. All runs are by Artificial Analysis. The v2 scores use the code-execution harness described above; the original CritPt column is the same organization's no-tools run on the original dataset, so the columns are not directly comparable. The page's headline named GPT-6 Astra at high effort, a configuration not shown in the table, as the top full-set score at 83.7%.^[8][7]

Sources: Artificial Analysis CritPt v2 and CritPt (legacy) pages, read 30 October 2026.^[8][7]

The CritPt GitHub README lists a v2 leaderboard that matches these full-set figures for the models it shares with the Artificial Analysis table. It also includes GPT-5.6 Sol at 74.9%, Gemini 4 Argon at 72.9% and GPT-5 at 28.9%.^[3]

Audits and criticism

Expert re-grading by Ansari et al.

In September 2026, Ali Ansari and 50 co-authors posted "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks" (arXiv:2609.13009; revised 17 October 2026). The paper audits six physics benchmarks, CritPt among them.^[12] Physics experts reviewed 56 of the 70 CritPt test challenges and found benchmark errors in 21. They repaired 19 and excluded 2, leaving a corrected set of 54. Because the official reference solutions are not public, the reviewers solved the problems independently to produce their own reference answers.^[12]

Source: Ansari et al., Table 1.^[12]

The two columns differ in more than question repairs. The corrected runs gave GPT-5.6 Sol and Fable 5 tool access through Codex and Claude Code, while Gemini 3.1 Pro used no tools. They also used an LLM judge comparing answers with the auditors' reference solutions, whereas the pre-audit scores came from the official grading server.^[12][14] The authors report that even before any problem statement was repaired, GPT-5.6 Sol (max) reached 82.59% mean accuracy on the 56 audited challenges under their procedure. They concluded that "most of the gap from the reported score reflects differences in evaluation rather than flaws in the questions".^[12] The paper also reports corrected CritPt scores for open models: Kimi K3 rose from 23.43% to 66.67%, GLM-5.3-Flash from 15.43% to 40.28%, and Qwen3.8-27B from 5.43% to 28.24%.^[12]

Anthropic's CritPT-Corrected

In its system card for Claude Fable 5.1 and Claude Mythos 5.1, dated 1 September 2026, Anthropic reported results on an internal version it calls CritPT-Corrected. Experts provided corrected versions of 31 problem statements "that were flagged due to underspecification, ambiguity, or typos". Anthropic reports mean pass@1 over 16 attempts at max effort with tools, with solutions graded by Claude Opus 4.8 acting as a judge against ground-truth solutions.^[13] Ansari et al. cite Fable 5.1's score on that version as 88.4% and note that the setups differ too much for their scores and Anthropic's to be directly comparable.^[12]

Epoch AI review

Epoch AI, which lists CritPt results sourced from the Artificial Analysis leaderboard, published a review of CritPt v1 dated 13 October 2026 and gave it the verdict "Flawed". Its threshold for that verdict is scoring defects affecting at least 20% of a benchmark. The review cites the Ansari et al. audit (at least 21 of 71 challenges, or 29.6%, flawed) and Anthropic's corrections to 31 of 71 (43.7%).^[14][15] Epoch also notes that the CritPT-Corrected gains cannot come from the fixes alone, since the fixes touched under half the challenges, and that it is unknown how much of the improvement came from tool use. The review covers v1 only and notes that the leaderboard reported v2 as of 22 October 2026.^[14]

Reception

CritPt's early scores were very low: in 2025 most models scored at or near zero on full challenges. The authors presented it as a measure of the "large disconnect between current model capabilities and realistic physics research demands".^[1] Over the following year, top scores on the original no-tools leaderboard rose into the 30% range, and audits argued that part of the remaining gap came from flawed questions and grading rather than model reasoning.^[7][12] With tools and the revised questions, the best full-set v2 scores were above 80% by late October 2026. The Artificial Analysis index now uses only the 40-problem hard subset, which leaves out the 30 problems that seven frontier models solved in every attempt.^[8][9]

Commenting on the Intelligence Index v5 results, physicist Steve Hsu wrote that the new math and physics evaluations exposed Chinese models "as far behind Astra and Opus 5.5". He attributed this to Chinese labs having focused on coding and agentic capabilities.^[17] On the CritPt v2 hard subset, the highest Chinese-developed model shown was Kimi K3.1 Pro at 45.0%, against 70.5% for GPT-6 Astra (max).^[8]

Continue reading

• Artificial AnalysisArtificial Analysis is an independent benchmarking and analytics platform that evaluates artificial intelligence models and API providers across intelligence, speed, price, and latency, and it is best known…Mentioned 20 times above
• SciCodeSciCode is a research coding benchmark that asks large language models to write Python code for realistic, PhD level scientific problems drawn from working scientists' day to day workflows.Mentioned twice above
• FrontierMathFrontierMath is an advanced mathematical reasoning benchmark created by Epoch AI in collaboration with over 60 expert mathematicians, including Fields Medalists Terence Tao, Timothy Gowers, and Richard…Listed in See also
• Humanity's Last ExamHumanity's Last Exam (HLE) is a multi-modal AI benchmark of 2,500 public expert-level academic questions (plus a private held-out set) spanning more than 100 disciplinesListed in See also
• GPQA DiamondGPQA Diamond is the 198-question hardest subset of the Graduate-Level Google-Proof Q&A Benchmark (GPQA), a set of PhD-level multiple-choice questions in biology, physics, and chemistry used to measure…Listed in See also
• AAI for ScienceAI for science is the use of artificial intelligence, particularly deep learning and large language models, to accelerate scientific discovery across biology, chemistry, physics, materials science…Listed in See also

More to read

• Benchmark (AI)
• Large Language Model
• GPT-6 Astra
• GPT-5
• Anthropic
• Epoch AI

Topics:AI BenchmarksAI for ScienceLarge Language ModelsModel Evaluation

References

1. ^1 ^2 ^3 ^4 ^5 ^6Zhu, M., Tian, M., Yang, X., et al. "Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark." arXiv:2509.26574 [cs.AI], submitted 30 September 2025, v4 revised 8 May 2026. arxiv.org/...2509.26574
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30Zhu, M., Tian, M., et al. "Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark" (full text, v4). arXiv HTML. arxiv.org/...2509.26574
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7CritPt-Benchmark. "Evaluation Pipeline for the CritPt Benchmark" (README). GitHub. github.com/...CritPt
4. ^1 ^2 ^3 ^4CritPt-Benchmark. Commit bc37a22, "CritPt v2", 22 October 2026. GitHub. github.com/...a22bedb8744dff8996ccce7f9eedec548915
5. ^CritPt-Benchmark/CritPt dataset. Hugging Face. huggingface.co/...CritPt
6. ^1 ^2CritPt. "News." critpt.com/news
7. ^1 ^2 ^3 ^4 ^5 ^6Artificial Analysis. "CritPt Benchmark" (legacy evaluation page). Accessed 30 October 2026. artificialanalysis.ai/...critpt
8. ^1 ^2 ^3 ^4 ^5 ^6Artificial Analysis. "CritPt v2 Benchmark." Accessed 30 October 2026. artificialanalysis.ai/...critpt-v2
9. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Artificial Analysis. "Intelligence Benchmarking Methodology" (CritPt, CritPt v2, CritPt v2 Hard Subset and version history sections). Accessed 30 October 2026. artificialanalysis.ai/...intelligence-benchmarking
10. ^Artificial Analysis (@ArtificialAnlys). "Announcing Artificial Analysis Intelligence Index v5..." X, 28 October 2026 (UTC). x.com/...2115265133467635877
11. ^1 ^2Zhu, M., et al. "Probing the Critical Point (CritPt) of AI Reasoning: Benchmarking LLMs at the Frontier of Physics." APS Global Physics Summit 2026, 17 March 2026. meetings-archive.aps.org/...mar-f42
12. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Ansari, A., Sun, H., Liu, A. Z., et al. "How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks." arXiv:2609.13009 [cs.AI], submitted 11 September 2026, v2 revised 17 October 2026. arxiv.org/...2609.13009
13. ^1 ^2Anthropic. "System Card: Claude Fable 5.1 & Claude Mythos 5.1," section 8.9 "CritPT-Corrected." 1 September 2026. www-cdn.anthropic.com/...205.1%20System%20Card.pdf
14. ^1 ^2 ^3 ^4 ^5Epoch AI. "CritPt v1" (Benchmark Review). Reviewed 13 October 2026. epoch.ai/...review
15. ^Epoch AI. "CritPt v1" (benchmark page). epoch.ai/...critpt
16. ^Tian, M., Gao, L., Zhang, S. D., et al. "SciCode: A Research Coding Benchmark Curated by Scientists." arXiv:2407.13168. arxiv.org/...2407.13168
17. ^Hsu, Steve (@hsu_steve). Post on Artificial Analysis Intelligence Index v5 rankings. X, 28 October 2026. x.com/...2115589096772919414
18. ^1 ^2Zhu, M., Tian, M., et al. "Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmark," v1. arXiv, 30 September 2025. arxiv.org/...2509.26574v1

Show all 18 references