VulcanBench

VulcanBench is an open-source benchmark for AI coding agents, created and run by Morgan Linton, co-founder and CTO of the apparel-sizing AI company Bold Metrics.^[1][2][3] It measures each model at every reasoning-effort level it offers, usually inside the vendor's own coding product such as Claude Code or OpenAI Codex, and reports token use, wall-clock time and estimated cost beside accuracy.^[1][4] Its harness and published task suites are on GitHub under the Apache 2.0 license.^[1][7] Results are published at vulcanbench.com and on Linton's X account, and have been picked up by the aggregator BenchLM and by news outlets including Crypto Briefing and Runtime Wire.^[8][9][10]

Overview

Origin

In a September 2026 blog post, Linton wrote that his engineering team at Bold Metrics was "on a path that was going to lead to a bill of over $300,000 for the year" for AI models, because engineers used each provider's best model "at High effort (or higher) for everything."^[2] He said evaluations built on the team's own codebase showed that many routine tasks did not need high effort or the newest frontier model, and that routing work across models and effort levels cut the bill to "under $30,000/year."^[2] Finding that most public benchmarks tested models only at maximum effort, he wrote his own tool, chose the name VulcanBench as "a lifelong Star Trek fan," and began posting results on X.^[2] He describes himself in the post as "not an expert" who "kinda fell into benchmarking."^[2] The site also sells private evaluation suites and model-routing recommendations built on a customer's own repositories, and accepts sponsorship through GitHub Sponsors.^[1]

Methodology

Frontier v4 tasks

Each Frontier v4 task gives the agent a compiled legacy binary, a written specification that has drifted from the binary's real behaviour, and a replacement implementation written to that specification. The agent can probe or disassemble the binary to recover the behaviour needed to fix the replacement, and hidden tests check the result.^[6] A candidate task is admitted only if, across three attempts each, GPT-5.6 Sol solves it at most once and Claude Opus 5 either needs a median of at least ten minutes or fails at least once; the reference solution must score 1.0 and the unmodified starting code 0.0 in repeated checks.^[6] The methodology page notes that these admission measurements are "not a guarantee of difficulty for later models."^[6]

Scoring

Functional correctness comes from deterministic hidden tests, with partial credit across test groups; a task counts as passed only with a functional score of 1.0, and a regression failure sets the functional score to zero.^[6] From September 2026 the combined score is 100 x (0.50F + 0.085Q + 0.085S + 0.33C), where F is functional correctness, Q a lint and complexity score from static tools (ruff and radon for Python), S a static security scan and C code quality.^[6] Reports published before September 2026 used a 50/15/15/20 weighting and were not rewritten.^[6]

Code quality is rated by a panel of two large language model judges, never including a model from the lab whose submission is graded, against a hash-frozen rubric written for "an engineer who has never seen the code" and must change it in one sitting.^[6][11] Each judge first takes a calibration exam on ten held-out programs. A reviewed score carries 24 of the 33 points and an intent-recovery probe, which checks whether the judge can find the task's documented specification departures from the code alone, carries 9; a planned third layer measuring follow-up maintenance had not been built as of October 30, 2026.^[11] For most Frontier v4 columns the judges are Muse Spark 1.3 and Grok 4.6; other panels are used where one of those judges would grade its own lab's model or was unavailable, and VulcanBench says scores from different panels are "not calibrated against each other."^[4][11] The methodology page states that these are model ratings "for a human reader," not human validation.^[6] See LLM-as-a-judge for the general technique.

Runs and cost

In the Frontier v4 sweeps each model runs once per task at each effort level, so a five-level sweep is 115 runs.^[4][11] Most columns ran on subscriptions (ChatGPT, Claude Max, Cursor), so the published cost per task is an API-equivalent estimate at list prices computed from the run's usage receipts, not a bill.^[4] The GPT-6.1 Terra report, for example, publishes per-run records, judge sub-scores and redacted traces, but withholds raw prompts and responses, submitted patches and the answer keys that describe hidden-test behaviour.^[11]

Leaderboard

As of the leaderboard update of October 30, 2026, Frontier v4 covered 17 models, 80 model and effort columns and 1,808 runs.^[4] The table shows each model's best-scoring column, for the seven models whose best column ranked in the top 18; the omitted ranks are other effort levels of the same models:

Source: VulcanBench Frontier v4 leaderboard, updated October 30, 2026.^[4] Ranks 2, 3 and 17 are Grok 4.7 at other effort levels, and ranks 7 and 9 to 15 are other levels of Claude Haiku 5.5, Opus 5.5, Fable 5.1 and Sonnet 5.5. Grok 4.7 was judged by Muse Spark 1.3 and GPT-6.1 Sol rather than Grok 4.6, so the leaderboard says its scores are "not strictly comparable" with the other columns; it has no list price, so no cost is shown. Fable 5.1's runs include 11 disclosed fallbacks to Claude Opus 4.8.^[4]

On the separate Routine v1 suite, 12 private tickets on small Python packages that test the cheapest adequate effort level, every one of the eight models tested passed all 12 tickets at its lowest effort level.^[4]

History

The retired SWE v3 board is mirrored by BenchLM, which as of October 29, 2026 gave it 3% of the reference weight in its coding leaderboard.^[8] VulcanBench says SWE v3 scores are not comparable with Frontier v4.^[4]

Limitations

VulcanBench is run by Linton, and its own pages list several limits. Results measure a model together with its harness, including system prompt, tools and any product routing, and effort labels "are not calibrated across vendors."^[6] Each cell is a single attempt per task, and the reported standard error describes task sampling, not judge uncertainty or repeated runs.^[11] A third of the combined score depends on model judges, and changes of judge panel make some columns not strictly comparable.^[4] The methodology page says contamination controls "cannot prove that a task was absent from training data."^[6]

Continue reading

• BBenchmark (AI)In artificial intelligence and machine learning, a benchmark is a specified evaluation used to compare systems under common conditions.Mentioned twice above
• LLLM-as-a-judgeLLM-as-a-judge is the practice of using a strong large language model to evaluate the outputs of other models, or of itself, in place of a human annotator.Listed in See also
• SSWE-benchSWE-bench is an execution-based benchmark for evaluating whether a language-model system can resolve real software issues.Listed in See also
• TTerminal-BenchTerminal-Bench is an open benchmark for evaluating AI agents on complex, real-world tasks performed through command-line terminal interfaces.Listed in See also
• Artificial AnalysisArtificial Analysis is an independent benchmarking and analytics platform that evaluates artificial intelligence models and API providers across intelligence, speed, price, and latency, and it is best known…Listed in See also
• Claude CodeClaude Code is an agentic software development product made by Anthropic. It uses models from the Claude family to inspect software projects, propose and apply changes, run development tools, and report…Mentioned 5 times above

More to read

• OpenAI Codex
• GPT-6.1 Terra
• SWE-bench Verified
• DeepSWE
• FrontierCode
• Pass@k

Topics:AI BenchmarksAI Code GenerationModel Evaluation

References

1. ^1 ^2 ^3 ^4 ^5VulcanBench. Home page. Accessed November 1, 2026. vulcanbench.com
2. ^1 ^2 ^3 ^4 ^5Linton, Morgan. "Why I started VulcanBench, and my journey building evals and benchmarking models across effort levels." VulcanBench, September 2, 2026. vulcanbench.com/...01-why-i-started-vulcanbench
3. ^1 ^2Bold Metrics. "About." Accessed November 1, 2026. boldmetrics.com/about
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15VulcanBench. "Leaderboard" (updated October 30, 2026). Accessed November 1, 2026. vulcanbench.com/leaderboard
5. ^1 ^2 ^3VulcanBench. "Benchmarks." Accessed November 1, 2026. vulcanbench.com/benchmarks
6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12VulcanBench. "Methodology." Accessed November 1, 2026. vulcanbench.com/methodology
7. ^1 ^2 ^3 ^4 ^5 ^6GitHub. "morganlinton/VulcanBench" (repository and releases). Accessed November 1, 2026. github.com/...VulcanBench
8. ^1 ^2BenchLM. "VulcanBench v3." Last updated October 29, 2026. benchlm.ai/...vulcanbench
9. ^1 ^2Crypto Briefing. "Grok 4.5 tops VulcanBench coding benchmark with 91% score, and AI investors should pay attention." July 19, 2026. cryptobriefing.com/...vulcanbench-coding-benchmark
10. ^1 ^2Merket, Ryan. "VulcanBench founder hits Muse Wonder's usage limits before finishing a benchmark." RuntimeWire, October 13, 2026. runtimewire.com/...wonder-vulcanbench-usage-limits
11. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Linton, Morgan. "GPT-6.1 Terra across every effort level." VulcanBench, October 30, 2026. vulcanbench.com/...swe-v4-gpt61-terra-v329
12. ^Linton, Morgan (@morganlinton). Post comparing GPT-6.1 Terra and Claude Sonnet 5.5 on VulcanBench. X, October 31, 2026. x.com/...2116602188244095037

Show all 12 references