SWE-bench Verified

SWE-bench Verified
Overview
Full name	Software Engineering Benchmark, Verified subset
Abbreviation	SWE-bench Verified
Description	A 500-task, human-validated subset of SWE-bench for evaluating AI agents on real-world GitHub issue resolution
Release date	2024-08-13
Latest version	1.0
Authors	OpenAI Preparedness team in collaboration with the Princeton SWE-bench team
Organization	OpenAI, Princeton University
Technical Details
Type	Code generation, bug fixing, software engineering
Modality	Text, code
Task format	GitHub issue resolution with unit-test grading
Number of tasks	500 (filtered from 1,699 candidates drawn from the original 2,294-instance SWE-bench)
Repositories	12 popular Python projects (Django, SymPy, scikit-learn, Sphinx, Matplotlib, pytest, xarray, astropy, pylint, requests, seaborn, Flask)
Evaluation metric	Resolve rate (% Resolved); FAIL_TO_PASS and PASS_TO_PASS test grading
Languages	Python
Performance
Initial baseline (Aug 2024)	33.2% (GPT-4o + Agentless)
Public SOTA	96.0% (Claude Opus 5, July 2026, self-reported)
Internal SOTA reported	95.5% (Claude Mythos 5, Anthropic, June 2026; restricted access)
Status	Deprecated by OpenAI for frontier evaluation on February 23, 2026
Resources
Website	Official page
Announcement	OpenAI blog
Deprecation post	OpenAI blog (Feb 2026)
Dataset	Hugging Face
Annotation guide	SWE-b Annotation Instructions (PDF)
License	MIT
Predecessor	SWE-bench

SWE-bench Verified is a 500-problem, human-validated subset of the SWE-bench software engineering benchmark, released on August 13, 2024 by OpenAI's Preparedness team together with the original Princeton SWE-bench authors. Each task asks an AI agent to read a real GitHub issue from one of 12 popular open-source Python repositories, edit the codebase, and produce a patch that is graded automatically against hidden unit tests. What sets Verified apart from the parent benchmark is that 93 software developers reviewed the candidate tasks (three annotators per task) to confirm that the problem statement is well specified, the unit tests do not reject valid solutions, and no other major issue, such as a broken environment, invalidates the task.^[1] From late 2024 through early 2026 it was the single most-cited coding benchmark in frontier model launches, until OpenAI itself deprecated it for frontier evaluation on February 23, 2026 over test flaws and training-data contamination.^[2][3]

Where the original SWE-bench's 2,294 instances were noisy enough that small score differences were hard to interpret, Verified produced a clean leaderboard signal that top labs raced to climb. The first published number was 33.2% for GPT-4o paired with the Agentless scaffold in August 2024, which roughly doubled Agentless's 16% on the full benchmark once the noisiest tasks were removed.^[1] The public state-of-the-art then rose to 49% by October 2024, past 80% in November 2025, to 88.6% for Claude Opus 4.8 in May 2026 and to a self-reported 96.0% for Claude Opus 5 in July 2026, with Anthropic's restricted Claude Mythos Preview posting 93.9% in April 2026, prompting widespread agreement that Verified had saturated.^[3][4][30] OpenAI's deprecation post argued that gains in this range were no longer meaningful: more than 59% of the hardest unsolved tasks in their audit had broken or unfair tests, and every frontier model tested could reproduce verbatim portions of the dataset.^[3]

Despite the deprecation, SWE-bench Verified remains widely used as a sanity check, an instructional benchmark, and a reference point against which newer evaluations like SWE-bench Pro, SWE-Lancer, and SWE-bench Multimodal are calibrated. Its design choices, especially the 93-annotator review process and the FAIL_TO_PASS / PASS_TO_PASS grading scheme, set the template that successor benchmarks have refined rather than replaced.^[1][3]

What is SWE-bench Verified used for?

SWE-bench Verified shares its core mechanics with the original SWE-bench: each task pairs a real GitHub issue with the codebase state immediately before the human-written fix was merged, plus a set of unit tests that distinguish the buggy state from the fixed state. An AI agent is given the issue text and the repository, must produce a code patch, and is graded on whether the patch flips the failing tests to passing without breaking the previously passing tests. What distinguishes Verified is the curation layer on top.

The core differences from SWE-bench are quality, scale, and intended use. The original benchmark sampled all 2,294 issue-PR pairs the team could mine from 12 repositories. Verified shrinks the set to 500 instances that an experienced human annotator confirmed are well-specified, fairly tested, and solvable. The result is a benchmark where a passing patch can be confidently interpreted as a real fix rather than a coincidence with a thin or buggy test suite.^[1]

Verified was released alongside the OpenAI Preparedness Framework's broader effort to measure dangerous autonomous capability in frontier models. The Preparedness team viewed software engineering ability as a leading indicator of model autonomy, and Verified was created so that the autonomy score would not be polluted by ambiguous tasks. The benchmark therefore plays a dual role in the AI safety and capabilities literature: it is both a standard product comparison and an input to OpenAI's risk assessments.^[1][2]

Origin and motivation

What problem did the original SWE-bench have?

When SWE-bench launched in October 2023, the headline result was that even the strongest available model, Claude 2 with BM25 retrieval, resolved only 1.96% of tasks. Through 2024, agent scaffolds like SWE-agent and commercial products like Devin drove that number above 13%, but the leaderboard was already showing strange behavior. Some tasks were trivially passable through patterns that did not actually fix the underlying bug. Others were essentially impossible because the test suite checked for behavior the issue had never specified.

A later analysis, SWE-Bench+ (Aleithan et al., October 2024), found that 32.67% of the successful SWE-agent + GPT-4 patches on the original benchmark involved solution leakage in the issue text or comments, and that a further 31.08% of passed patches were suspicious because the tests were too weak to verify correctness. After filtering, the SWE-agent + GPT-4 score dropped from 12.47% to 3.97%, a roughly threefold reduction. Independent reviewers had also flagged tests that rejected functionally correct solutions because they checked for the exact wording or formatting used in the gold patch.^[1][7]

OpenAI's Preparedness team needed a more reliable measurement to feed into its Model Autonomy risk evaluations. The team also wanted a benchmark whose score could be cited in product launches without each lab having to caveat that the underlying tasks were noisy. Both goals pointed in the same direction: human-validate a subset of SWE-bench and publish a clean signal.

Who created SWE-bench Verified and when?

OpenAI Preparedness collaborated with the authors of the original SWE-bench (Princeton's Carlos E. Jimenez and John Yang are among the credited authors of the Verified post) on a curation effort that shipped on August 13, 2024. OpenAI's announcement identified three problems to fix: unit tests that "are often overly specific, and in some cases are even unrelated to the issue," issue descriptions that are underspecified, and development environments that are "sometimes difficult to reliably set up."^[1] The post summarized the curation process: "We worked with 93 software developers experienced in Python to manually screen SWE-bench samples for quality." The result was "a subset of the original test set from SWE-bench, consisting of 500 samples verified to be non-problematic by our human annotators."^[1]

Verified inherits the upstream benchmark's MIT license and uses the same Docker harness and grading scheme, so existing SWE-bench tooling continues to work. The two teams also coordinated on the official princeton-nlp/SWE-bench_Verified Hugging Face dataset and on hosting a separate Verified leaderboard at swebench.com.^[1][2]

Methodology

Annotator pool

OpenAI contracted 93 software developers experienced in Python to perform the annotation. Each prospective annotator had to pass onboarding tests built from 50 samples that OpenAI engineers had hand-labeled. Annotators were not necessarily prior experts in the relevant codebases but were given time to familiarize themselves with each one.^[1]

Sampling and triple review

From the 2,294-instance SWE-bench test set, OpenAI selected 1,699 random samples to label. Every sample was reviewed by three independent annotators using a structured rubric. The triple-review design was meant to surface disagreement on borderline cases without leaning on any single reviewer's judgment.^[1][2]

Annotation rubric

Annotators labeled each task on two criteria scored on a 0 to 3 severity scale, where 0 and 1 are minor and 2 or 3 mean the sample should be discarded, plus a free-form flag for other major issues. A difficulty estimate was also collected but was not used for filtering:^[1][8]

Alongside the discard rubric, annotators estimated how long an experienced developer would need to decide on and implement the fix, given the cleaned issue text. Those estimates feed the difficulty distribution discussed below.^[1][2]

Filtering outcomes

The annotation pass produced striking quality numbers. About 38.3% of the 1,699 candidates were flagged for underspecified problem statements, and 61.1% were flagged for unit tests that may unfairly mark valid solutions as incorrect. In total, roughly 68.3% of the original SWE-bench samples were judged inadequate and removed. The remaining 500 instances form SWE-bench Verified.^[1][3][7]

Difficulty distribution

Using the annotators' time estimates, the SWE-bench Verified team published a difficulty breakdown. Most tasks are short, which has been both a feature (cheap to evaluate) and a criticism (does not capture multi-day projects).^[1][9]

A companion analysis by Epoch AI confirmed the skew, finding that around 90% of Verified tasks are fixes an experienced engineer could complete in under an hour. Quick fixes (under 15 minutes) average about 5 changed lines of code, the 15-minute-to-1-hour bucket averages 14, and the 1-4 hour bucket about 50.^[10]

How does SWE-bench Verified differ from the original SWE-bench?

Verified inherits the construction pipeline of SWE-bench but differs in size, quality, and reporting role. The table below summarizes the key differences.

The core mechanical difference is curation. Same repositories, same harness, same FAIL_TO_PASS / PASS_TO_PASS grading. What changed was the confidence one could place in any given score. On Verified, an 80% resolve rate plausibly means the agent solved 400 problems whose fixes a panel of reviewers agreed on. On the full benchmark, an 80% resolve rate would have included tasks where the test suite let multiple non-equivalent patches pass, or where the issue text omitted information the agent had to infer from the gold patch.

Repository and task structure

Like the parent benchmark, Verified draws from 12 Python open-source repositories. Each task includes the issue title and body, the codebase at the pre-fix commit, the dependency manifest, and the FAIL_TO_PASS and PASS_TO_PASS test sets needed to grade a candidate patch.

Django dominates because it has both a long history of high-quality issue tracking and a large, well-tested codebase that supplies many candidate tasks. The five largest repositories account for about 83% of the dataset and the five smallest for about 8%.^[10][31] Epoch AI has criticized the skew because a handful of repositories, Django above all, dominate the benchmark.^[10]

Evaluation framework

Containerized execution

Verified uses the same Docker harness as the broader SWE-bench family. Each task is associated with a specific environment image that fixes the Python version and the exact dependency versions present at the issue's commit. This containerization is what lets different labs compare scores: without pinned environments, NumPy or pytest version drift alone could swing scores by several percentage points.^[2][11]

How is a task graded?

A task is marked resolved if and only if every FAIL_TO_PASS test passes after applying the agent's patch and every PASS_TO_PASS test continues to pass. A patch that fixes the bug but breaks an unrelated regression test counts as a failure. A patch that does not apply cleanly because of whitespace drift also counts as a failure. The strictness of the grading is part of why Verified produces interpretable signals: an agent that achieves 80% has met both the bug-fix and the no-regression bars on 400 separate tasks.^[1][2][11]

Reporting conventions

The primary metric is % Resolved, the share of the 500 tasks for which the agent's patch passes the grading rule. Secondary metrics that became common in the late-2024 to early-2026 reporting cycle include:

• Pass@k: success rate when the agent gets k independent attempts.
• Cost per resolved task: dollar cost of the model and tools required to resolve a task on average.
• Wall clock per task: median seconds the agent runs before producing a patch.
• Variance across reruns: standard deviation of resolve rate across 3 to 5 evaluation seeds, since stochastic agents can swing several points run-to-run.

From mid-2025 onward, several leaderboards (notably swebench.com, llm-stats.com, and Scale's labs.scale.com) added cost columns next to the headline accuracy figure, partly in response to criticism that high-cost reasoning agents were inflating reported scores without delivering proportional value.^[4][5][12]

Historical leaderboard

Verified scores climbed faster than almost any other coding benchmark. The progression below tracks the public state-of-the-art at major checkpoints from launch through 2026.

The trajectory looks like a textbook capability ramp. From August 2024 to May 2026, the public state-of-the-art among generally-available models rose from roughly 33% to 88.6%, an increase of more than 55 percentage points in under two years. Three forces drove the gain: stronger base models (Claude 3.5 to 4.8, GPT-4o to GPT-5.3), better reasoning (extended thinking, chain-of-thought, self-verification), and more capable agent scaffolds (Agentless, SWE-agent, OpenHands, Claude Code).

What is the highest SWE-bench Verified score?

As of November 2026, the highest self-reported score by a generally available model is 96.0%, posted by Anthropic's Claude Opus 5 (July 2026), ahead of Claude Fable 5 (95.0%, June 2026).^[30] Before the Claude 5 generation, the top mark was 88.6% from Claude Opus 4.8 (released May 28, 2026), which edged out the April 2026 mark of 87.6% set by Claude Opus 4.7.^[29][30] Anthropic's restricted models are reported higher still: Claude Mythos Preview at 93.9% (April 2026) and Claude Mythos 5 at 95.5% (June 2026), neither of which is generally available.^[30] The public top of the leaderboard is dominated by Anthropic: the steel.dev aggregation snapshot (updated June 12, 2026) lists Claude Opus 4.8 (88.6%), Claude Opus 4.7 (87.6%), Claude Opus 4.5 (80.9%), and Claude Opus 4.6 (80.8%) above the strongest non-Anthropic entries, which cluster near 80% (Gemini 3.1 Pro 80.6%, MiniMax M2.5 80.2%, GPT-5.2 80.0%).^[30] Because OpenAI deprecated Verified for frontier evaluation in February 2026, these figures should be read as the closing state of the benchmark's competitive era rather than a live race.

Cognition Devin and the original SWE-bench

Before Verified existed, Cognition Labs' Devin helped make SWE-bench a marketing benchmark. In March 2024 Cognition reported that Devin resolved 13.86% of issues on a random 25% subset (570 of 2,294) of the original full benchmark, against a previous unassisted best of 1.96%.^[14] Cognition's report predates Verified and contains no Verified results. By August 5, 2024, shortly before Verified launched, the top agents on the original leaderboard scored 20% on the full SWE-bench and 43% on SWE-bench Lite.^[1]

Top-15 snapshot (April 2026)

The table below is a snapshot of the public SWE-bench Verified leaderboard from April 2026, sourced from the public swebench.com leaderboard, llm-stats.com aggregation, and lab-reported scores. Internal-only models like Claude Mythos are excluded.^[4][5]

In May 2026, after this snapshot, Anthropic released Claude Opus 4.8, which posted 88.6% on Verified and 69.2% on the recommended successor SWE-bench Pro, taking the top generally-available position.^[29][30]

Top scoring approaches and frameworks

The agent scaffolds that dominated the Verified leaderboard tended to combine an interactive shell or editor with a search tool, a test runner, and some form of self-verification. Several reference designs are worth singling out.

Agentless

Agentless, developed by researchers at the University of Illinois Urbana-Champaign, eschews an autonomous agent loop in favor of a fixed three-stage pipeline: localize the bug, repair it, and validate the fix. OpenAI tested GPT-4o with several open-source scaffolds, and Agentless gave the best result, the original 33.2% Verified baseline. The cleaned task set roughly doubled Agentless's 16% score on the full SWE-bench, and the scaffold remained the most popular reference for cheap, reproducible Verified runs through 2025.^[1][16]

SWE-agent

SWE-agent, the official Princeton agent paper published at NeurIPS 2024, introduced the Agent-Computer Interface (ACI): a curated set of shell commands that make repository navigation and editing easier for language models. SWE-agent's scaffolds powered many of the open-source Verified entries, including the Live-SWE-agent variant that reached 79.2% with Claude Opus 4.5 in late 2025.^[17]

OpenHands (formerly OpenDevin)

OpenHands consolidated several research scaffolds into a community-driven open-source platform supporting browser, shell, and editor tools. It became the de facto open-source counterweight to closed agent products like Devin and was widely used in academic Verified submissions. The framework supports multiple LLM backends and ships with reusable plugins for retrieval, multi-attempt sampling, and verifier-based selection.^[18]

Aider

Aider, Paul Gauthier's open-source command-line coding assistant, publishes its own polyglot code-editing benchmark rather than Verified results. It is mostly used as an interactive tool, but its public benchmark page played an outsized role in popularizing cost-adjusted reporting and the practice of comparing the same scaffold across many models.^[19]

Claude Code

Claude Code, Anthropic's terminal-based agent introduced in early 2025, became the reference scaffold for Anthropic's own Verified numbers. Its bash tool, file editor, and computer-use tool let the underlying Claude model do most of the heavy lifting with a relatively shallow surrounding harness. Claude Code's design influenced how other labs approached agent engineering, especially the move toward giving the model more direct shell access rather than wrapping it in heavy intermediation.^[20]

RA-Aid, Moatless, and AutoCodeRover

A second tier of agents has held steady positions in the mid-leaderboard. RA-Aid is a research agent built on top of OpenHands with a focus on retrieval-augmented planning. Moatless Tools is a lightweight scaffold popular with budget-conscious researchers. AutoCodeRover, developed at the National University of Singapore, uses spectrum-based fault localization to narrow the search space before patch generation. Each contributed important ideas (decoupled localization, deterministic patches, coverage-guided edits) that later commercial agents absorbed.^[21][22]

Other agents commonly seen on the leaderboard

Patterns common to the strongest submissions

Independent of any specific framework, the highest-scoring Verified entries through 2025 and 2026 tended to share a few patterns. They run multiple candidate patches and use a verifier (often an LLM-as-judge with regression testing) to pick the best one. They allow generous tool use (file search, AST queries, language servers, test runners) instead of restricting the agent to a fixed action set. They feed the test output back into the model in a tight loop so the agent can self-correct. And they explicitly handle whitespace and formatting drift in patch generation, which avoided the silent grading failures that plagued earlier submissions.

Why did OpenAI deprecate SWE-bench Verified?

On February 23, 2026, OpenAI's Mia Glaese and Olivia Watkins published "Why SWE-bench Verified no longer measures frontier coding capabilities." The post argued that the benchmark had reached the end of its useful life as a frontier evaluation and recommended SWE-bench Pro as the new community standard. OpenAI's core conclusion was blunt: "improvements on SWE-bench Verified no longer reflect meaningful improvements in models' real-world software development abilities," and "instead, they increasingly reflect how much the model was exposed to the benchmark at training time."^[3]

Findings cited in the deprecation post

OpenAI's case rested on two findings, prompted by the observation that state-of-the-art progress had slowed, from 74.9% to 80.9% over the preceding six months:^[3]

1. Broken specifications dominate the unsolved tail. OpenAI audited 138 problems (27.6% of the dataset) that OpenAI o3 did not consistently solve over 64 independent runs, each reviewed by at least six experienced engineers. It found that 59.4% of them had material issues in test design or problem description: 35.5% of the audited tasks had narrow tests that enforce specific implementation details, 18.8% had wide tests that check for functionality the issue never specified, and 5.1% had miscellaneous issues.^[3][23]
2. Training-data contamination is now pervasive. Every frontier model OpenAI tested could reproduce verbatim portions of the gold patch or problem statement when probed with a task ID, in most cases alongside a short hint from the task. Models implicated in the audit included GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash. In one example, GPT-5.2's chain-of-thought traces revealed knowledge of unspecified test requirements that could only have come from training data exposure.^[3]

Recommendation

OpenAI recommended migrating to SWE-bench Pro, which uses 1,865 longer tasks across more diverse public, held-out, and commercial codebases. Pro tasks are explicitly chosen to require 1 to 4 hours of human effort or more, in contrast with Verified's bias toward sub-hour fixes. Initial Pro scores from frontier models ran roughly 20 or more percentage points below their Verified scores; OpenAI's stated reason for the switch was that Pro "empirically seems to suffer less from contamination issues." The gap held into mid-2026: Claude Opus 4.8 scored 88.6% on Verified but only 69.2% on SWE-bench Pro.^[3][24][29]

Industry response

The deprecation post was discussed extensively on the Latent Space podcast in the same week, with Glaese and Watkins arguing that the move was about measurement validity rather than a critique of Verified's design. Anthropic, Google DeepMind, and Meta did not immediately stop reporting Verified numbers, but most subsequent product launches paired the Verified score with a Pro score. Several outlets (CodeSOTA, blockchain.news, Latent Space, marc0.dev) wrote retrospectives within weeks framing the deprecation as a generational shift in how the field measures coding ability.^{[3][6][23][24][25]}

Successors and complementary benchmarks

OpenAI's deprecation accelerated work on a family of successor benchmarks that aim to restore measurement validity. The most influential are summarized below.

SWE-Lancer

OpenAI launched SWE-Lancer in February 2025 to evaluate models on real freelance software engineering tasks from Upwork, largely posted for the Expensify open-source repository, with end-to-end tests triple-verified by experienced software engineers. The benchmark covers more than 1,400 tasks with payouts totaling roughly $1 million in real dollars, ranging from $50 bug fixes to $32,000 feature implementations. Tasks split between independent contributor (IC) work and managerial decisions over technical proposals. Initial reported scores were modest: on the public Diamond split, Claude 3.5 Sonnet completed 26.2% of IC tasks and 44.9% of managerial tasks, while OpenAI's o1 (high reasoning effort) reached 16.5% and 41.5% respectively. The dollars-earned metric resonated outside the academic community and made SWE-Lancer popular with industry analysts.^[26]

SWE-bench Multimodal

Introduced in October 2024 (arXiv:2410.03859), SWE-bench Multimodal extends the harness to JavaScript repositories with image-bearing issues. The dataset contains 619 task instances drawn from 17 user-facing JavaScript repositories, with 862 images embedded across the problem statements. Image categories include code screenshots (194 instances), diagrams (107), error messages (54), digital art (38), maps (35), and data visualizations (28). Multimodal scores have lagged Verified scores by a wide margin because the task requires both visual grounding and code reasoning, and JavaScript test frameworks (Jest, Mocha, Playwright) make patch validation more involved than the Python set.^[27]

Limitations

Test suite weaknesses

OpenAI's deprecation audit confirmed earlier independent findings that Verified's test suites are not always trustworthy. An empirical study published as arXiv:2503.15223 examined patches from three leading issue-solving tools and found that weaknesses in SWE-bench's patch validation let 7.8% of patches count as correct while failing the developer-written test suite. Its PatchDiff differential-testing technique found behavioral differences from the ground-truth patch in 29.6% of plausible patches, and the authors estimated that these weaknesses inflate reported resolution rates by 6.2 percentage points. OpenAI's own audit reported even higher rates of broken specifications among the hardest unsolved tasks.^[3][28]

Data contamination

Verified inherits the parent benchmark's contamination problem. More than 94% of SWE-bench issues were filed before the knowledge cutoff dates of major pre-trained language models. OpenAI's February 2026 study demonstrated that frontier models could regenerate parts of the gold patch or problem statement when probed with a task ID and short hints. Even SWE-bench Live's monthly refresh did not entirely solve the problem because tasks rotate into model training data on a similar timescale.^[3][7]

Repository selection bias

The 12 source repositories are all open-source Python projects with strong test cultures. Many real-world codebases have sparse tests, proprietary dependencies, or architectural patterns not represented in this set. As a result, high Verified scores do not necessarily predict performance on arbitrary production codebases. The Django dominance in particular gives the dataset a web framework flavor that is over-represented compared with the broader software ecosystem.^[9][10]

Task-length skew

Epoch AI's analysis showed that the majority of Verified tasks are relatively simple. About 91% can be completed by a human in under one hour and 39.2% in under 15 minutes. The benchmark therefore primarily measures an agent's ability to fix straightforward bugs rather than tackle architectural changes or large feature implementations. This skew is the single biggest reason successor benchmarks like SWE-bench Pro and SWE-Lancer reset the difficulty floor.^[9][10]

Python-only coverage

Verified is Python-only. Performance on Verified does not reliably generalize to JavaScript, Java, C++, Go, or Rust. SWE-bench Multilingual, Multi-SWE-bench, and SWE-bench Live partially address this gap, but Verified's outsized leaderboard role meant the field's headline metric was Python-bound through early 2026.^[3][27]

Cost and reproducibility

A full Verified evaluation is resource-intensive: the SWE-bench Docker guide calls for at least 120 GB of free disk space, 8 CPUs, and 16 GB of RAM. Cloud evaluation removes the local hardware burden but introduces direct dollar cost, and reasoning-heavy agents add substantial inference costs on top.^[11]

Saturation

The most consequential limitation is saturation. When OpenAI deprecated the benchmark in February 2026, the state of the art was 80.9% and OpenAI had found flawed tests in many of the remaining unsolved tasks; by mid-2026, generally available Claude 5 models self-reported 95% to 96% and the restricted Claude Mythos Preview 93.9%, leaving little meaningful headroom.^[30] Verified can still discriminate between weaker models, but the lab competition that defined the late-2024 to late-2025 era has effectively ended.^[3]

Continue reading

• SSWE-benchSWE-bench is an execution-based benchmark for evaluating whether a language-model system can resolve real software issues.Mentioned 76 times above
• Claude (language model)Claude is a family of large language models (LLMs) developed by Anthropic, an American artificial intelligence safety and research company.Mentioned 52 times above
• OpenAIOpenAI is an American artificial intelligence research and deployment organization.Mentioned 49 times above
• BBenchmark (AI)In artificial intelligence and machine learning, a benchmark is a specified evaluation used to compare systems under common conditions.Mentioned 31 times above
• AnthropicAnthropic is an American artificial intelligence (AI) safety and research company founded in 2021 by Dario Amodei, Daniela Amodei, and other former OpenAI researchers, best known for the Claude family of large…Mentioned 28 times above
• SWE-agentSWE-agent is an open-source autonomous software engineering agent created by the Princeton NLP group (Princeton Language and Intelligence, with co-authors from Stanford) and first released on April 2, 2024.Mentioned 8 times above

More from See also

• Global-MMLU
• FormulaOne
• MMLU-Redux
• PutnamBench
• ScienceAgentBench
• HumanEval
• LiveCodeBench
• Devin (AI software engineer)
• Aider
• Cursor (code editor)
• Claude Code
• GPT-4
• AI Code Generation

Topics:AI BenchmarksAI Code GenerationModel Evaluation

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18OpenAI. (August 13, 2024). "Introducing SWE-bench Verified." openai.com/...introducing-swe-bench-verified
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7SWE-bench Verified official page. swebench.com/verified
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15OpenAI. (February 23, 2026). "Why SWE-bench Verified no longer measures frontier coding capabilities." Mia Glaese and Olivia Watkins. openai.com/...o-longer-evaluate-swe-bench-verified
4. ^1 ^2 ^3llm-stats.com. SWE-bench Verified leaderboard (April 2026). llm-stats.com/...swe-bench-verified
5. ^1 ^2Marco Patzelt. (April 2026). "SWE-Bench Verified Leaderboard, Claude Opus 4.7 Leads." marc0.dev/...leaderboard
6. ^1 ^2 ^3TokenMix. (April 2026). "SWE-Bench 2026: Claude Opus 4.7 Wins 87.6% vs GPT-5.3 85.0%." tokenmix.ai/...swe-bench-2026-claude-opus-4-7-wins
7. ^1 ^2 ^3Aleithan, R., Xue, H., Mohajer, M. M., Nnorom, E., Uddin, G., and Wang, S. (October 2024). "SWE-Bench+: Enhanced Coding Benchmark for LLMs." arXiv:2410.06992. arxiv.org/...2410.06992
8. ^OpenAI. (2024). "SWE-b Annotation Instructions." Annotator-facing PDF. cdn.openai.com/...swe-b-annotation-instructions.pdf
9. ^1 ^2 ^3Epoch AI. "SWE-bench Verified." epoch.ai/...swe-bench-verified
10. ^1 ^2 ^3 ^4 ^5Epoch AI. (2024). "What skills does SWE-bench Verified evaluate?" epoch.ai/...kills-does-swe-bench-verified-evaluate
11. ^1 ^2 ^3SWE-bench Docker setup and cloud evaluation guide. swebench.com/...docker_setup
12. ^Scale Labs. SWE-Bench Pro Leaderboard (Public). labs.scale.com/...swe_bench_pro_public
13. ^Anthropic. (October 2024). "Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet." anthropic.com/...swe-bench-sonnet
14. ^Cognition Labs. (2024). "SWE-bench Technical Report." cognition.ai/...swe-bench-technical-report
15. SWE-bench leaderboard archive. swebench.com
16. ^Xia, C. S., Deng, Y., Dunn, S., and Zhang, L. (2024). "Agentless: Demystifying LLM-based Software Engineering Agents." arXiv:2407.01489. arxiv.org/...2407.01489
17. ^Yang, J. et al. (2024). "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." NeurIPS 2024. swe-agent.com
18. ^OpenHands (formerly OpenDevin) project page. github.com/...OpenHands
19. ^Aider benchmarks. aider.chat/...leaderboards
20. ^Anthropic. (2025). Claude Code documentation. docs.anthropic.com/...claude-code
21. ^AutoCodeRover. "Autonomous Program Improvement." arXiv:2404.05427.
22. ^Moatless Tools repository. github.com/...moatless-tools
23. ^1 ^2blockchain.news. (2026). "OpenAI Abandons SWE-bench Verified After Finding 59% of Failed Tests Were Flawed." blockchain.news/...fied-contamination-flawed-tests
24. ^1 ^2Latent Space. (2026). "The End of SWE-Bench Verified, with Mia Glaese & Olivia Watkins." latent.space/...swe-bench-dead
25. ^CodeSOTA. (2026). "Is SWE-bench Verified Contaminated? OpenAI Shifts to SWE-bench Pro." codesota.com/...swe-bench-contamination-debate
26. ^OpenAI. (February 2025). "Introducing the SWE-Lancer Benchmark." openai.com/...swe-lancer
27. ^1 ^2Yang, J. et al. (2024). "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?" arXiv:2410.03859.
28. ^"Are 'Solved Issues' in SWE-bench Really Solved Correctly? An Empirical Study." arXiv:2503.15223.
29. ^1 ^2 ^3MacRumors. (May 28, 2026). "Anthropic Launches Claude Opus 4.8 With Gains in Coding and Honesty." macrumors.com/...anthropic-claude-opus-4-8
30. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Steel.dev. "SWE-bench Verified Leaderboard 2026: Latest Coding Agent Scores" (snapshot updated June 12, 2026; page last updated September 4, 2026, accessed November 2026). leaderboard.steel.dev/...swe-bench-verified
31. ^princeton-nlp/SWE-bench_Verified dataset (test split statistics). Hugging Face. huggingface.co/...SWE-bench_Verified
32. ^1 ^2Anthropic. (February 24, 2025). "Claude 3.7 Sonnet and Claude Code." anthropic.com/...claude-3-7-sonnet
33. ^Anthropic. (May 22, 2025). "Introducing Claude 4." anthropic.com/...claude-4
34. ^OpenAI. (August 7, 2025). "Introducing GPT-5." openai.com/...introducing-gpt-5

Show all 34 references

External links

• SWE-bench Verified official page
• Introducing SWE-bench Verified (OpenAI, August 2024)
• Why SWE-bench Verified no longer measures frontier coding capabilities (OpenAI, February 2026)
• SWE-bench Verified dataset on Hugging Face
• SWE-b Annotation Instructions (PDF)
• SWE-Bench Verified Leaderboard (llm-stats.com)
• Epoch AI SWE-bench Verified analysis
• SWE-agent project
• Latent Space, The End of SWE-Bench Verified