# VulcanBench

> Source: https://aiwiki.ai/wiki/vulcanbench
> Updated: 2026-10-31
> Fact-checked: 2026-10-31
> Categories: AI Benchmarks, AI Code Generation, Model Evaluation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "VulcanBench." aiwiki.ai, 31 Oct 2026. https://aiwiki.ai/wiki/vulcanbench
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

VulcanBench is an open-source benchmark for AI coding agents, created and run by Morgan Linton, co-founder and CTO of the apparel-sizing AI company Bold Metrics.[1][2][3] It measures each model at every reasoning-effort level it offers, usually inside the vendor's own coding product such as [Claude Code](https://aiwiki.ai/wiki/claude_code) or [OpenAI Codex](https://aiwiki.ai/wiki/openai_codex), and reports token use, wall-clock time and estimated cost beside accuracy.[1][4] Its harness and published task suites are on GitHub under the Apache 2.0 license.[1][7] Results are published at vulcanbench.com and on Linton's X account, and have been picked up by the aggregator BenchLM and by news outlets including Crypto Briefing and Runtime Wire.[8][9][10]

## Overview

| Property | Details |
| --- | --- |
| Maintainer | Morgan Linton[1][3] |
| First release | v0.1.0 on GitHub, June 20, 2026[7] |
| Current suite | VulcanBench Frontier v4 (23 behavioural-reconstruction tasks)[4] |
| Other suites | Routine v1, Safety v1, Verdict v1 and v2; SWE v3 retired in August 2026[4][5] |
| Scoring (Frontier v4) | 50% functional correctness, 8.5% lint and complexity, 8.5% security, 33% code quality[4][6] |
| License | Apache 2.0[7] |
| Repository | github.com/morganlinton/VulcanBench (95 stars as of November 1, 2026)[7] |

## Origin

In a September 2026 blog post, Linton wrote that his engineering team at Bold Metrics was "on a path that was going to lead to a bill of over $300,000 for the year" for AI models, because engineers used each provider's best model "at High effort (or higher) for everything."[2] He said evaluations built on the team's own codebase showed that many routine tasks did not need high effort or the newest frontier model, and that routing work across models and effort levels cut the bill to "under $30,000/year."[2] Finding that most public benchmarks tested models only at maximum effort, he wrote his own tool, chose the name VulcanBench as "a lifelong Star Trek fan," and began posting results on X.[2] He describes himself in the post as "not an expert" who "kinda fell into benchmarking."[2] The site also sells private evaluation suites and model-routing recommendations built on a customer's own repositories, and accepts sponsorship through GitHub Sponsors.[1]

## Methodology

### Frontier v4 tasks

Each Frontier v4 task gives the agent a compiled legacy binary, a written specification that has drifted from the binary's real behaviour, and a replacement implementation written to that specification. The agent can probe or disassemble the binary to recover the behaviour needed to fix the replacement, and hidden tests check the result.[6] A candidate task is admitted only if, across three attempts each, GPT-5.6 Sol solves it at most once and [Claude Opus 5](https://aiwiki.ai/wiki/claude_opus_5) either needs a median of at least ten minutes or fails at least once; the reference solution must score 1.0 and the unmodified starting code 0.0 in repeated checks.[6] The methodology page notes that these admission measurements are "not a guarantee of difficulty for later models."[6]

### Scoring

Functional correctness comes from deterministic hidden tests, with partial credit across test groups; a task counts as passed only with a functional score of 1.0, and a regression failure sets the functional score to zero.[6] From September 2026 the combined score is 100 x (0.50F + 0.085Q + 0.085S + 0.33C), where F is functional correctness, Q a lint and complexity score from static tools (ruff and radon for Python), S a static security scan and C code quality.[6] Reports published before September 2026 used a 50/15/15/20 weighting and were not rewritten.[6]

Code quality is rated by a panel of two [large language model](https://aiwiki.ai/wiki/large_language_model) judges, never including a model from the lab whose submission is graded, against a hash-frozen rubric written for "an engineer who has never seen the code" and must change it in one sitting.[6][11] Each judge first takes a calibration exam on ten held-out programs. A reviewed score carries 24 of the 33 points and an intent-recovery probe, which checks whether the judge can find the task's documented specification departures from the code alone, carries 9; a planned third layer measuring follow-up maintenance had not been built as of October 30, 2026.[11] For most Frontier v4 columns the judges are Muse Spark 1.3 and [Grok 4.6](https://aiwiki.ai/wiki/grok_4_6); other panels are used where one of those judges would grade its own lab's model or was unavailable, and VulcanBench says scores from different panels are "not calibrated against each other."[4][11] The methodology page states that these are model ratings "for a human reader," not human validation.[6] See [LLM-as-a-judge](https://aiwiki.ai/wiki/llm_as_a_judge) for the general technique.

### Runs and cost

In the Frontier v4 sweeps each model runs once per task at each effort level, so a five-level sweep is 115 runs.[4][11] Most columns ran on subscriptions (ChatGPT, Claude Max, Cursor), so the published cost per task is an API-equivalent estimate at list prices computed from the run's usage receipts, not a bill.[4] The GPT-6.1 Terra report, for example, publishes per-run records, judge sub-scores and redacted traces, but withholds raw prompts and responses, submitted patches and the answer keys that describe hidden-test behaviour.[11]

## Leaderboard

As of the leaderboard update of October 30, 2026, Frontier v4 covered 17 models, 80 model and effort columns and 1,808 runs.[4] The table shows each model's best-scoring column, for the seven models whose best column ranked in the top 18; the omitted ranks are other effort levels of the same models:

| Rank | Model and harness | Effort | Combined | Tasks passed | $/task |
| ---: | --- | --- | ---: | ---: | ---: |
| 1 | [Grok 4.7](https://aiwiki.ai/wiki/grok_4_7) (Cursor) | Extra-high | 93.15 | 23/23 | Unavailable |
| 4 | [Claude Haiku 5.5](https://aiwiki.ai/wiki/claude_haiku_5_5) (Claude Code) | Max | 92.16 | 22/23 | $1.54 |
| 5 | [Claude Sonnet 5.5](https://aiwiki.ai/wiki/claude_sonnet_5_5) (Claude Code) | Max | 92.02 | 23/23 | $5.52 |
| 6 | [Claude Fable 5.1](https://aiwiki.ai/wiki/claude_fable_5_1) (Claude Code) | Max | 91.84 | 23/23 | $9.06 |
| 8 | [Claude Opus 5.5](https://aiwiki.ai/wiki/claude_opus_5_5) (Claude Code) | High | 91.11 | 22/23[11] | $3.27 |
| 16 | [GPT-6.1 Terra](https://aiwiki.ai/wiki/gpt_6_1_terra) (Codex) | Max | 90.00 | 23/23 | $0.36 |
| 18 | [GPT-6 Astra](https://aiwiki.ai/wiki/gpt_6_astra) (Codex) | Max | 89.30 | 23/23 | $2.57 |

Source: VulcanBench Frontier v4 leaderboard, updated October 30, 2026.[4] Ranks 2, 3 and 17 are Grok 4.7 at other effort levels, and ranks 7 and 9 to 15 are other levels of Claude Haiku 5.5, Opus 5.5, Fable 5.1 and Sonnet 5.5. Grok 4.7 was judged by Muse Spark 1.3 and [GPT-6.1 Sol](https://aiwiki.ai/wiki/gpt_6_1_sol) rather than Grok 4.6, so the leaderboard says its scores are "not strictly comparable" with the other columns; it has no list price, so no cost is shown. Fable 5.1's runs include 11 disclosed fallbacks to Claude Opus 4.8.[4]

On the separate Routine v1 suite, 12 private tickets on small Python packages that test the cheapest adequate effort level, every one of the eight models tested passed all 12 tickets at its lowest effort level.[4]

## History

| Date | Event |
| --- | --- |
| June 20, 2026 | v0.1.0 released on GitHub[7] |
| June 30 to July 1, 2026 | First reports compare Claude Sonnet 5 and Claude Opus 4.8 across effort levels[5] |
| July 2026 | SWE v3 suite (23 tasks from real merged open-source pull requests) begins; on July 12 Grok 4.5 leads at 91%[5] |
| July 19, 2026 | Crypto Briefing reports Grok 4.5's 91.3% on VulcanBench[9] |
| August 2026 | SWE v3 retired; its final board (15 models, 42 columns) frozen[4] |
| September 9, 2026 | First results under the September 2026 code-quality protocol (GPT-6 Astra and Claude Fable 5.1 rescored)[6] |
| October 13, 2026 | Runtime Wire reports that Linton exhausted Muse Code's weekly usage allowance before finishing a run of Meta's [Muse Wonder](https://aiwiki.ai/wiki/muse_wonder)[10] |
| October 30, 2026 | GPT-6.1 Terra report published; leaderboard updated; v0.10.0 released[4][7][11] |
| October 31, 2026 | Linton posts a GPT-6.1 Terra versus Claude Sonnet 5.5 comparison on X[12] |

The retired SWE v3 board is mirrored by BenchLM, which as of October 29, 2026 gave it 3% of the reference weight in its coding leaderboard.[8] VulcanBench says SWE v3 scores are not comparable with Frontier v4.[4]

## Limitations

VulcanBench is run by Linton, and its own pages list several limits. Results measure a model together with its harness, including system prompt, tools and any product routing, and effort labels "are not calibrated across vendors."[6] Each cell is a single attempt per task, and the reported standard error describes task sampling, not judge uncertainty or repeated runs.[11] A third of the combined score depends on model judges, and changes of judge panel make some columns not strictly comparable.[4] The methodology page says contamination controls "cannot prove that a task was absent from training data."[6]

## See also

- [AI benchmark](https://aiwiki.ai/wiki/benchmark)
- [SWE-bench](https://aiwiki.ai/wiki/swe_bench)
- [Terminal-Bench](https://aiwiki.ai/wiki/terminal_bench)
- [Artificial Analysis](https://aiwiki.ai/wiki/artificial_analysis)
- [LLM-as-a-judge](https://aiwiki.ai/wiki/llm_as_a_judge)

## References

1. VulcanBench. Home page. Accessed November 1, 2026. https://vulcanbench.com/
2. Linton, Morgan. "Why I started VulcanBench, and my journey building evals and benchmarking models across effort levels." VulcanBench, September 2, 2026. https://vulcanbench.com/blog/01-why-i-started-vulcanbench.html
3. Bold Metrics. "About." Accessed November 1, 2026. https://boldmetrics.com/about
4. VulcanBench. "Leaderboard" (updated October 30, 2026). Accessed November 1, 2026. https://vulcanbench.com/leaderboard.html
5. VulcanBench. "Benchmarks." Accessed November 1, 2026. https://vulcanbench.com/benchmarks.html
6. VulcanBench. "Methodology." Accessed November 1, 2026. https://vulcanbench.com/methodology.html
7. GitHub. "morganlinton/VulcanBench" (repository and releases). Accessed November 1, 2026. https://github.com/morganlinton/VulcanBench
8. BenchLM. "VulcanBench v3." Last updated October 29, 2026. https://benchlm.ai/benchmarks/vulcanbench
9. Crypto Briefing. "Grok 4.5 tops VulcanBench coding benchmark with 91% score, and AI investors should pay attention." July 19, 2026. https://cryptobriefing.com/grok-4-5-tops-vulcanbench-coding-benchmark/
10. Merket, Ryan. "VulcanBench founder hits Muse Wonder's usage limits before finishing a benchmark." RuntimeWire, October 13, 2026. https://runtimewire.com/article/muse-wonder-vulcanbench-usage-limits
11. Linton, Morgan. "GPT-6.1 Terra across every effort level." VulcanBench, October 30, 2026. https://vulcanbench.com/benchmarks/swe-v4-gpt61-terra-v329.html
12. Linton, Morgan (@morganlinton). Post comparing GPT-6.1 Terra and Claude Sonnet 5.5 on VulcanBench. X, October 31, 2026. https://x.com/morganlinton/status/2116602188244095037