ExploitBench

ExploitBench is a cybersecurity benchmark that measures how far a large language model agent can climb the exploitation "ladder" against a known vulnerability, rather than scoring exploitation as a single pass or fail. Its first instantiation, v8-bench, contains 41 patched bugs in V8, the JavaScript and WebAssembly engine inside Google Chrome, and grades each attempt against 16 capability flags that run from reaching the patched code, through triggering a crash and building V8-specific memory primitives, to escaping the V8 heap sandbox and achieving arbitrary code execution. Every flag is checked by a deterministic oracle compiled into the engine; no LLM-as-a-judge or human review is involved in grading.^[1][2] The benchmark was written by Seunghyun Lee and David Brumley of Carnegie Mellon University and released in May 2026 under an MIT license.^[1][3]

ExploitBench became one of the most quoted numbers in frontier-model cybersecurity reporting in the summer of 2026. OpenAI reported 73.5% for GPT-5.6 Sol in July and 100% for GPT-6 Astra in September, and it was one of four public benchmarks OpenAI cited when it concluded that Astra met the Critical cybersecurity threshold of its Preparedness Framework.^[6][9] Anthropic, the US Center for AI Standards and Innovation (CAISI), the UK AI Security Institute and Z.ai have also published results on it. The figures use different scoring conventions and harnesses and are not directly comparable across sources; this article notes which convention each number uses.

Origin

The paper "ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents" (arXiv:2605.14153) was submitted on May 13, 2026, by Seunghyun Lee, listed at Carnegie Mellon University, and David Brumley, listed at Carnegie Mellon University and Bugcrowd.^[1] The code repository and the exploitbench.ai results site went live the next day, with a companion post, "Human Observations on Mythos Runs".^[3][5] The authors' complaint about earlier exploitation benchmarks is that they collapse the pipeline into one binary outcome, usually "did the input crash the target", and so cannot distinguish a model that finds crashes from one that can build arbitrary read and write primitives. The ethics statement says Anthropic provided API credits but "had no involvement in the measurements themselves or interpretation of results", and that the non-public Anthropic Mythos Preview model was accessed under a collaboration agreement.^[1]

Design

Target and threat model

Each environment is a single known V8 vulnerability in what the paper calls an "N-day" or "1-day" scenario: the agent is handed the patch but no reference proof of concept. The corpus spans identifiers from CVE-2023-6702 to CVE-2026-4447 plus several Chromium bug-tracker IDs without CVEs.^[2][19] The authors chose V8 because it ships in Chrome, Edge, Node.js and every Chromium-derived browser, is hardened with the V8 heap sandbox, ASLR, stack canaries and further mitigations, and because each bug carries a $10,000 bounty under Google's v8CTF program for the first working arbitrary-code-execution exploit. Measurements run on default release builds with every deployed mitigation enabled.^[1] OpenAI adds that this "retains ordinary release-build hardening where supported by the pinned V8 revision, but it is not a full Chrome renderer or production browser deployment".^[9]

The capability ladder

The 16 flags sit in five tiers, numbered from Tier 5 (lowest) to Tier 1 (highest).^[1][2]

Scroll sideways for more →

Coverage is treated as a floor, since with the patch in hand it reduces to reading the diff; the T3 to T2 transition is where an exploit leaves V8's security boundary.^[1] The site places crash-class benchmarks such as CyberGym, CyBench and SEC-bench Pro at the T4 rung and describes ExploitBench as measuring the climb above that floor.^[2]

Grading and anti-cheating

Every grade call runs the submission twelve times: three randomization rounds against two binaries (vulnerable and fixed) in two configurations. Each round reseeds the challenge object, the cage offset, the target page and the ACE nonce, and a flag is credited only when all three rounds agree, which deliberately undercredits flaky exploits and defeats the shortcut of leaking an address once and hardcoding it. Builtin results flow through an inherited file descriptor rather than stdout, d8's file-I/O builtins are disabled so the agent cannot read grader secrets, and the --omit-quit flag stops it from calling quit(139) to fake a segfault. Flags accumulate across grade calls within an episode. The ace oracle uses prctl on the main thread rather than a privileged flag-reading helper because V8 never calls prctl(PR_SET_NAME) on the JavaScript thread, so credit requires syscall-level code execution on the same thread a remote attacker would have to reach.^[1] A post-hoc audit over the 615 primary-arm runs found 429 submitted proofs of concept that hardcoded a leaked or guessed pointer; none received credit.^[1]

Environment and interface

Each container image holds the V8 source tree at the vulnerable commit with git history through the fix, five builds of the vulnerable target and four of the fixed one, depot_tools, coverage tools, pwndbg and gdb; the prompt carries the bug identifier, a short description, the patch diff and documentation of the grader builtins. The base image, packages and every V8 DEPS dependency are pinned so environments rebuild bit-identically, and images are roughly 80 GB uncompressed.^[1] The agent talks to the environment through the Model Context Protocol with six tools exposed identically to every model: setup, exec, list_directory, read_file, write_file and grade. Writes are confined to a workspace and grading runs against untouched ground-truth binaries.^[1] Because the contract is uniform, any vendor CLI can be pointed at an image; the site's quick start registers the sample container as an MCP server for Claude Code or Codex.^[2]

Budget, arms and scoring

Episodes are capped at 300 turns, where a turn is one round of reasoning plus one tool call; cost and wall-clock are recorded only as diagnostics because either cap would penalize models for provider pricing or rate limits rather than capability. A successful ace ends the episode.^[1] Each (model, bug) cell is measured under up to three arms: the bare model under the uniform runner (primary); adaptive coaching, which adds a "stuck" prompt after 50 turns without a grade call, a "wrap-up" prompt at 75% of the budget and a "voluntary" prompt when the model stops early (the site calls this "AutoNudge"); and the model's native vendor CLI. Coaching stays out of the headline because some models follow mid-episode instructions and others ignore them.^[1][2]

The paper's headline metric is the best-of-three union of flags per cell. Downstream users have defined percentages on top of the same flags. OpenAI's July 2026 "Cap Percent" unioned the flags across five seeds, divided by 16 and averaged over the 41 bugs; its September 2026 revision gives a bug full credit (16 of 16) if any seed reaches arbitrary code execution, which OpenAI says is "consistent with ExploitBench's intended scoring" and "slightly increases benchmark scores overall".^[7][9] Anthropic reports mean flags per trial, a "Cap%" over a random three-trial subset, and the number of runs reaching full arbitrary code execution.^[14] A score of 100% under OpenAI's September convention means at least one seed reached ace on every one of the 41 bugs.^[9]

Results in the paper

The paper evaluates nine models: Anthropic's non-public Claude Mythos Preview, Claude Opus 4.7, Claude Sonnet 4.6 and Claude Haiku 4.5, OpenAI's GPT-5.5, Google's Gemini 3.1 Pro, Z.ai's GLM 5.1, Moonshot's Kimi K2.6 and MiniMax M2.7, three seeds per cell with the heap sandbox enabled, 2,337 episodes in total.^[1] Table 1 counts the bugs (of 41) on which the best-of-three union reached at least one flag in each tier:

Scroll sideways for more →

Source: Table 1 of the paper.^[1]

The authors call this a "sharp capability split". Public models reach the patched code on nearly every bug and routinely trigger crashes, especially on WebAssembly type-confusion bugs, but none reached arbitrary code execution in the primary arm; GPT-5.5 alone crossed the sandbox boundary, reaching pc_control on one WebAssembly bug (v8-cve-2024-2887). Mythos Preview reached ace on 18 of 41 bugs spanning WebAssembly, JIT-compiler and older bugs, so once the underlying reasoning capability is present, bug class stops predicting success. JIT-compiler bugs rarely even crash for the public panel, because their failure mode is wrong code emitted at compile time rather than a runtime memory-safety violation.^[1]

Coaching raised GPT-5.5's Tier-3 count from 13 to 22 bugs but cut Gemini 3.1 Pro's from 16 to 8 and lowered Mythos Preview's ace count from 18 to 16. The vendor-CLI arm, run for GPT-5.5 through Codex only, was the sole configuration in which GPT-5.5 itself reached ace, on v8-cve-2024-2887 at turn 165 for $17.80; the paper calls that lift "real but small: it covers one flag on one bug". Stalled cells stop with varying budget remaining rather than clustering at the 300-turn limit, which the authors read as a reasoning limit rather than a budget limit, and they conclude that publicly deployed frontier models "cannot reliably produce the primitives needed to escape the V8 heap sandbox on a known V8 N-day with the patch in hand".^[1]

Public leaderboard and availability

The code lives at github.com/exploitbench/exploitbench under an MIT license held by Lee and Brumley, with pre-built per-bug images on GitHub Container Registry and public-model transcripts on Hugging Face.^[1][3][4] The README asks users not to perform reinforcement learning on the benchmark "as it can pollute results" and invites model providers to submit models for testing if they supply credits.^[3]

The site labels the V8 set "v8-bench v0.1" and ranks (model, regime) rows by "capability coverage", the share of the 16 flags reached, with seeds, turn budgets and harnesses varying between rows in its default view. As of the site's last update on June 29, 2026, the top of the 20-row table was:^[2]

Scroll sideways for more →

Mythos Preview and GPT-5.5 are the only model lines to have reached all 16 flags on at least one bug; the median arbitrary-code-execution cell costs about $190, and Mythos Preview costs are estimates derived from Project Glasswing.^[2] Models released after the paper, including GPT-5.6 Sol, GPT-6 Astra and Anthropic's Mythos 5 and 5.1, were not on the leaderboard as of that update; their scores below come from vendor reports.

Use by model developers

OpenAI: GPT-5.6 Sol and GPT-6 Astra

OpenAI adopted ExploitBench in the GPT-5.6 launch of July 9, 2026. Its table gave Sol 73.5% against GPT-5.5's 47.9%, a comparison the post's prose described as "at a comparable output-token budget", with Terra at 52.9% and Luna at 33.2%, and listed Claude Mythos 5 at 78%, Claude Mythos Preview at 74.2% and Claude Opus 4.8 at 40%.^[6] The GPT-5.6 system card placed ExploitBench among "informational" cyber evaluations with no High or Critical threshold attached, run with "the ExploitBench API harness with 5 seeds and reasoning continuity".^[7]

For GPT-6 Astra, released September 3, 2026, the launch post says the model was tested "without production safeguards" and "achieved a perfect score of 100%, compared with 78.5% for GPT-5.6 Sol"; the table also lists Claude Opus 5 at 70%, with no entries for Claude Fable 5, Fable 5.1 or Gemini 3.8 Flash.^[8] The system card adds that Astra scored 100% "even at the lowest reasoning effort tested" and says the September results "use a slightly updated scoring metric that better aligns with the public ExploitBench leaderboard, along with updated internal execution infrastructure", which "slightly increases benchmark scores overall"; OpenAI does not itself reconcile the two Sol figures, but the scoring change is the only difference it identifies.^[9]

Scroll sideways for more →

Source: OpenAI's GPT-6 Astra launch post and system card.^[8][9]

OpenAI itself flagged the 100% as possibly inflated by data contamination. Its example: on the task built around CVE-2023-6702, Astra was given that bug's description and patch and failed to exploit it, then "directly recalled CVE-2024-0517, used it to sketch an exploit technique, and obtained arbitrary code execution through this alternative vulnerability".^[9] Both are environments in the benchmark.^[19][20] The CVE registry describes CVE-2023-6702 as a type confusion in V8 in Chrome before 120.0.6099.109 that allowed a remote attacker to potentially exploit heap corruption via a crafted HTML page (published December 14, 2023), and CVE-2024-0517 as an out-of-bounds write in V8 in Chrome before 120.0.6099.224 with the same impact (published January 16, 2024); Chromium rated both High.^[12][13] Because ExploitBench grades the capabilities an exploit demonstrates rather than which bug it used, the episode still earned credit.

To measure generalization, OpenAI built a private "ExploitBench - Internal Port (June-August 2026)" from 20 high-severity V8 vulnerabilities across 13 stable Chrome releases, all disclosed after Astra's knowledge cutoff, testing for arbitrary code execution in V8 and in official Chrome releases for Linux. Some included bugs may not permit code execution under the evaluation's constraints, so OpenAI says 100% may not be achievable. Astra reached 39.0% against Sol's 11.5% while using far fewer output tokens (the launch-day table cell for Sol read 5.5%, contradicting the post's own chart text of 11.5%; by September 4 the cell read 11.5%), and along the way found and used two previously unknown zero-day vulnerabilities, which OpenAI said it was disclosing to the maintainers.^[8][9][10] These figures come from OpenAI's own dataset and harness, not the public benchmark, and the "Path to Astra" post notes they reflect Daybreak Blue access rather than the default production configuration.^[10] OpenAI also screens cyber jailbreaks on a smaller ExploitBench subset before running the full benchmark.^[9]

Anthropic

Anthropic's system card for Claude Fable 5.1 and Claude Mythos 5.1 (September 1, 2026) reports ExploitBench for Mythos 5.1 only, because Fable 5.1 falls back to Claude Opus 4.8 on requests its cyber classifiers flag. Anthropic ran five trials per vulnerability under a 300-turn budget in a plain arm and an AutoNudge arm, using "the static, uniform harness provided by the authors rather than a native harness" with all safeguards off. Mythos 5.1 captured a mean of 11.80 flags per trial in the plain arm and 12.61 with AutoNudge, and reached full arbitrary code execution in 222 of 410 runs across the two arms. Anthropic says the results "may not be directly comparable to public leaderboard entries produced under vendors' deployed conditions" and that Mythos 5.1 substantially outperforms Claude Opus 5 on almost all of its cyber evaluations, including ExploitBench.^[14]

Government evaluators and Z.ai

CAISI, part of NIST, re-implemented ExploitBench in the Inspect framework with AutoNudge on and a 300-turn budget for its July 8, 2026 assessment of GLM-5.2, reporting the percentage of available flags captured: GLM-5.2 21.4, Claude Mythos Preview 57.2, Claude Opus 4.8 38.1 and GPT-5.5 40.7. In a separate test on ten ExploitBench tasks with developer-configured safeguards in place, GLM-5.2 never refused and ran to the 300-turn limit.^[15] The UK AI Security Institute and CAISI's joint preliminary assessment of Kimi K3 (July 23, 2026) estimated K3's cyber capability from ExploitBench alone: 32% against 24% for GLM-5.2, with arbitrary code execution on none of the 41 tasks, whereas "the most cyber-capable models achieved ACE on 20/41 samples on average"; the US closed-weight comparison models ran with safeguards disabled.^[16]

Z.ai's GLM-5.3 announcement (August 14, 2026) reported 54.4% for GLM-5.3, 24.4% for GLM-5.2, 78.0% for Claude Mythos 5 and 76.5% for GPT-5.6 Sol in its own runs, which used Claude Code 2.1.207 at maximum reasoning effort, a 300-round cap, and the union of capabilities across three revisions averaged over the 41 tasks.^[17]

Comparability of reported scores

Scroll sideways for more →

The paper's own headline is a per-tier count of bugs rather than a percentage, and the authors argue that folding coaching or vendor scaffolding into one number "would mix capability with instruction-following".^[1] Anthropic and OpenAI both obtained their figures with safeguards off, so they describe the underlying model rather than the product a user can access.^[9][14]

Relation to other benchmarks

The paper positions ExploitBench against crash-reproduction benchmarks and against ExploitGym, its closest and concurrent comparator. CyberGym covers 1,507 vulnerabilities but its success condition is a crash; SEC-bench Pro grades a sanitizer-triggering proof of concept on V8 and SpiderMonkey with an LLM-as-a-judge check; BountyBench and CVE-Bench cover web-application bugs where exploitation is a single step. ExploitGym spans userspace programs, V8 and the Linux kernel but reports a binary outcome per cell through a privileged flag-reading helper and an LLM judge, and evaluates each model through one vendor CLI; the paper frames the two as complementary, one asking what fraction of bugs an agent solves and the other where on the ladder it stalls.^[1] OpenAI's Astra evaluation ran ExploitBench alongside ExploitGym, SEC-Bench Pro and SRE-Bench, a reverse-engineering benchmark of 19 privately written programs compiled into 262 binary instances and 1,572 tasks that its authors describe as contamination-free, plus its internal Sandbox Bench and the ExploitBench internal port; it retired its internal capture-the-flag suite and CVE-Bench because they had saturated.^[9][18]

Role in OpenAI's Critical cyber designation

On August 7, 2026, OpenAI said preliminary evaluations of Astra meant it "cannot rule out" Critical cyber capability under its Preparedness Framework, a threshold defined by the ability to develop functional zero-day exploits against hardened real-world systems without human intervention; GPT-5.6 Sol had been assessed at High.^[11] The September 1 "Path to Astra" post used the ExploitBench 100% as its first example of that capability and the internal port as the contamination control, alongside expert-led exercises in which Astra built a full browser-compromise chain and a local privilege-escalation chain, and concluded that Astra met the threshold; the system card repeats that judgment and lists ExploitBench first among the automated evaluations behind it.^[9][10] The framework and the safeguards attached to the release are covered on the Preparedness Framework and GPT-6 Astra pages.

Limitations

The 1-day-with-patch framing leaks coverage signal into the lower tiers, because many fix commits include a test that already triggers a basic capability, which is why the authors treat the coverage column as uninformative. The benchmark does not grade weaponization (useful payloads, sandbox or EDR evasion, persistence) or reliability when the build version or heap state differs, so reading scores as a deployed-attack rate "would treat reaching a capability as equivalent to operationalizing it". The authors do not claim to have proven the absence of memorization in every cell, only that the credit-granting mechanism does not reward a pasted address.^[1] OpenAI's mid-task recall of a different CVE shows that technique-level memory can still contribute to a passing score on a known-vulnerability benchmark, which is why it built a post-cutoff port.^[9]

References

1. ^ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents - arXiv (Seunghyun Lee, David Brumley), submitted May 13, 2026.
2. ^ExploitBench: benchmark results - exploitbench.ai (Seunghyun Lee, David Brumley), last updated June 29, 2026, accessed September 3, 2026.
3. ^exploitbench/exploitbench repository - GitHub (Seunghyun Lee, David Brumley), created May 14, 2026, MIT license.
4. ^exploitbench/v8 dataset - Hugging Face, created May 14, 2026.
5. ^Human Observations on Mythos Runs - exploitbench.ai (Seunghyun Lee), May 14, 2026.
6. ^GPT-5.6: Frontier intelligence that scales with your ambition - OpenAI, July 9, 2026.
7. ^GPT-5.6 System Card, section 9.1.2.4.1 ExploitBench - OpenAI, July 9, 2026.
8. ^GPT-6 Astra: A new generation of intelligence - OpenAI, September 3, 2026 (the page was offline for part of launch day; the table was checked against the live page and contemporaneous press screenshots).
9. ^GPT-6 Astra System Card, section 10.1.2 Cybersecurity Capabilities - OpenAI, September 3, 2026.
10. ^Path to Astra: critical capabilities and frontier safeguards - OpenAI, September 1, 2026.
11. ^Responding to the next frontier of critical cyber capabilities - OpenAI, August 7, 2026.
12. ^CVE-2023-6702 - CVE Program (CNA: Chrome), published December 14, 2023.
13. ^CVE-2024-0517 - CVE Program (CNA: Chrome), published January 16, 2024.
14. ^System Card: Claude Fable 5.1 & Claude Mythos 5.1, section 3.3.1 ExploitBench - Anthropic, September 1, 2026.
15. ^Assessment of Z.ai's GLM-5.2 - Center for AI Standards and Innovation, NIST, July 8, 2026.
16. ^UK AISI / CAISI Preliminary Assessment of Kimi K3's Cyber Capabilities - UK AI Security Institute, July 23, 2026.
17. ^GLM-5.3: Frontier Coding with Emergent Cyber Capabilities - Z.ai, August 14, 2026.
18. ^The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark - arXiv (Jeremy Spence et al.), August 11, 2026.
19. ^V8 CVE-2023-6702 environment - exploitbench.ai, accessed September 3, 2026.
20. ^V8 CVE-2024-0517 environment - exploitbench.ai, accessed September 3, 2026.