SWE-bench

SWE-bench is an execution-based benchmark for evaluating whether a language-model system can resolve real software issues. Each task provides a repository at a specified commit and the text of a resolved issue; the system must produce a code patch that passes tests for the requested fix without breaking selected existing behavior. The original benchmark contains 2,294 tasks drawn from 12 open-source Python repositories. Carlos E. Jimenez and six coauthors introduced it in 2023, and the paper appeared at ICLR 2024.^[1] A SWE-bench score therefore measures an evaluated system - including its model, agent scaffold, tools, context-selection method, and evaluation setup - rather than a language model in isolation.

Task construction

The original authors mined merged pull requests from 12 established Python repositories. A candidate pull request had to resolve at least one GitHub issue and change test files. The construction pipeline then compared tests before and after applying the pull request. At least one test had to change from failing to passing, while other selected tests had to keep passing. After filtering an initial collection of more than 90,000 pull requests, the process produced 2,294 benchmark instances.^[1]

An instance identifies the repository, the base commit, the issue-derived problem statement, the reference patch, and a patch containing tests. It also records two test groups: FAIL_TO_PASS, for tests expected to fail before the fix and pass afterward, and PASS_TO_PASS, for regression tests expected to pass both before and after the fix. The reference patch and test patch are evaluator artifacts; they are not a standard solver's answer or prompt. The official dataset documentation also records fields such as an instance identifier, repository version, issue URL, pull-request URL, and creation date.^[2]

This design makes the unit of evaluation larger than a short code-completion exercise. A task can require locating relevant code in a repository, understanding behavior across functions or files, editing the working tree, and returning an applicable diff. It does not, however, reproduce every part of professional software development: the benchmark normally ends when its selected tests pass, not after code review, deployment, monitoring, or long-term maintenance.^[1]

Original dataset profile

The original test set is unevenly distributed across its 12 repositories. Django contributes 850 of the 2,294 tasks, while Flask contributes 11. Repository-level differences can therefore materially affect an aggregate score.

The paper reports an average issue length of 195.1 words and an average non-test codebase size of about 438,000 lines. Human reference patches edited an average of 32.8 lines, 1.7 files, and 3 functions. These are means with wide ranges: the largest reference patch edited 5,888 lines and 31 files. Such variation is one reason that a single resolution percentage conceals substantial differences in task type and effort.^[1]

The repository mix also limits external validity. All 12 original projects are open-source Python packages, and several are scientific-computing or developer-tool projects. Results do not automatically transfer to private services, mobile applications, embedded systems, security-sensitive code, or projects written in other languages. Later variants expanded language and repository coverage, but they are separate evaluations rather than retroactive changes to the original set.^[1]

Evaluation protocol

For each prediction, the evaluation harness creates a containerized repository environment, applies the proposed patch, and runs the task's tests. A task is resolved only when the patch applies and all required FAIL_TO_PASS and PASS_TO_PASS tests pass. The principal aggregate metric is the percentage of evaluated instances resolved.^[1][3]

The two test groups serve different purposes:

• FAIL_TO_PASS tests are intended to verify the new behavior associated with the issue.
• PASS_TO_PASS tests are intended to detect regressions in behavior that already worked.
• A patch that fixes the target test but breaks a selected regression test is not resolved.
• A malformed or non-applicable patch cannot be counted as resolved even if its intended edit appears plausible.

The official evaluation documentation specifies a prediction record containing instance_id, model_name_or_path, and model_patch. It also reports counts for submitted, completed, resolved, empty, and errored predictions. Consequently, a published result should state its denominator and how incomplete or errored instances were handled, not only a percentage.^[4][3]

The tests are a programmatic oracle, not a proof that a patch is the only correct implementation. A correct alternative can be rejected if a hidden test enforces an unstated detail, while an incomplete patch can pass if coverage is weak. Environment configuration can also change outcomes. These limitations became a major subject of later audits.

What the score combines

An issue-resolution run generally has at least three stages: locating relevant code, proposing edits, and validating or revising those edits. A system may expose the model to a fixed retrieval result, let an agent search the repository interactively, or allow the agent to run tests and inspect failures. These are materially different experimental conditions even when they use the same model checkpoint.

The standard resolved metric is binary at task level. It does not award partial credit for finding the right file, identifying the defect, writing a nearly correct patch, or passing some but not all required tests. That simplicity makes aggregation straightforward, but it also means that two systems with the same resolution rate can fail for very different reasons. Task-level logs are needed to distinguish localization failures, malformed patches, timeouts, environment failures, and semantically wrong solutions.

Repeated sampling adds another distinction. Pass@1 describes one attempt per task. Pass@k or a best-of-k procedure gives a system multiple chances and normally yields a higher probability of at least one passing patch. Neither statistic is inherently improper, but they answer different questions and have different compute costs. A comparison that omits the attempt count can be misleading.

Core datasets and variants

Several datasets use the SWE-bench name but differ in task count, repository coverage, modality, or curation. Their scores are not interchangeable.

The SWE-bench Lite test set was sampled after excluding, among other cases, tasks with images, external hyperlinks, edits to more than one file, large reference patches, file creation or deletion, and certain error-message tests. Those choices reduce evaluation cost but also change the task distribution. Lite is therefore not merely a cheaper run of the full set.^[5]

OpenAI and the SWE-bench authors created SWE-bench Verified after a 2024 annotation campaign. Ninety-three developers experienced with Python reviewed 1,699 randomly selected original tasks, with three independent labels per task. OpenAI reported that 38.3% of reviewed samples were flagged for underspecified problem statements and 61.1% for tests that could unfairly reject valid solutions; 68.3% were removed for these or other issues. The released subset contains 500 tasks, including 196 estimated to require less than 15 minutes and 45 estimated to require more than one hour for an experienced developer.^[6]

For its release analysis, OpenAI evaluated the dated gpt-4o-2024-05-13 model with several open-source scaffolds. It reported 33.2% resolved for the best of those configurations, Agentless, using a single seed and the authors' closest-documented or default settings. That number is useful as a release-era baseline, but the source itself cautions that it may differ from official leaderboard submissions.^[6]

The published SWE-bench Multimodal paper has an internal count discrepancy. Its abstract reports 617 tasks, but the dataset-construction section says that annotators removed 24 impossible tasks to leave 619, divided into 102 development and 517 test tasks. The latter split sums to 619. Reporting the split explicitly avoids silently choosing between the paper's conflicting totals.^[7]

SWE-bench Multilingual contains 300 manually verified tasks from 42 repositories. It covers C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust while retaining the issue-to-patch evaluation pattern.^[8] These variants test different distributions, so a result should always name the exact dataset and split.

Original baselines

The original paper evaluated then-current language models using two ways of selecting repository context. A BM25 retriever selected files based on lexical relevance to the issue text. An "oracle" condition instead supplied files changed by the human reference patch. The oracle condition was an analysis tool, not a realistic deployment setting, because it used knowledge of which files the accepted solution edited.^[1]

With BM25 retrieval, the paper reported a 1.96% resolution rate for Claude 2 on the full 2,294-task set. This result established that the original task was difficult for the evaluated 2023-era setup. It is a historical baseline, not a current leaderboard value and not directly comparable with later results on Lite, Verified, or other variants. Even within one dataset, changing file retrieval, prompting, tools, context limits, patch format, or the number of attempts changes the evaluated system.^[1]

The retrieval experiments illustrate this dependency. At a 27,000-token limit, BM25 retrieved at least one file changed by the reference patch for about half of the instances, but it retrieved none of those files for nearly the other half. Supplying more retrieved context did not consistently improve resolution. The paper attributed part of this behavior to difficulty localizing the relevant code amid distracting context.^[1]

The authors also trained SWE-Llama, based on 7-billion- and 13-billion-parameter Code Llama models, as a repository-editing baseline. They collected about 19,000 issue and pull-request pairs from 37 repositories that did not overlap the test repositories. After excluding training sequences longer than 30,000 tokens, about 10,000 instances remained for fine-tuning. These models were research baselines associated with the original paper, not additional SWE-bench test sets.^[1]

Leaderboards and reproducibility

The official leaderboards accept results for several SWE-bench variants. Their submission page points to a public experiments repository that stores official submissions and directs evaluators to the current harness or cloud tooling. Multimodal evaluation has a separate submission path because its test data are not distributed in the same way as the ordinary text-only sets.^[9]

A leaderboard row should be interpreted as a configuration, not as a timeless property of a named model. At minimum, a reproducible report should identify:

• the exact dataset, split, and dataset revision;
• the model identifier or dated API version;
• the agent scaffold and its version;
• allowed tools, repository access, retrieval method, and image input where applicable;
• the evaluation-harness version and environment;
• the number of attempts per task and whether the statistic is pass@1, pass@k, or another estimator;
• token, time, and monetary budgets;
• the number of submitted, completed, errored, and resolved tasks; and
• prediction files or trajectories when release is permitted.

Comparisons that change several of these variables at once cannot isolate an improvement in the underlying large language model. Likewise, the highest live leaderboard entry can change after an article is written. A durable encyclopedia article is better served by explaining the protocol and linking to the maintained leaderboard than by copying a fast-aging ranking table.

Containerization improves consistency but does not remove every source of drift. A container build can depend on package registries, archived dependencies, base images, CPU architecture, or network access. Harness updates can repair task specifications or environment scripts. For reproducibility, a result should preserve the predictions, task-level logs, container or image identifiers, and exact harness revision in addition to the headline score.^[3][9]

Confidence also depends on sample size and task dependence. A 300-task subset has wider sampling uncertainty than a 2,294-task set, and tasks from the same repository are not necessarily independent. A repository-weighted analysis may answer a different question from a task-weighted average dominated by Django. Reporting per-repository results and uncertainty can reveal whether an apparent gain is broad or concentrated in a few projects.

Data-quality studies

The SWE-Bench+ preprint examined 251 patches that SWE-Agent with GPT-4 had successfully submitted on the full benchmark and that the authors confirmed as passing the recorded tests. Among this conditional sample of successful patches, the researchers classified 82 patches, or 32.67%, as cases where the issue or comments disclosed the solution. They classified another 78, or 31.08%, as incorrect, incomplete, or misplaced fixes that passed weak tests. These percentages describe the reviewed successful-patch sample; they are not estimates that the same fractions of all 2,294 tasks are defective.^[10]

Verified was designed to reduce underspecification and test mismatch, but later evidence showed that manual filtering did not eliminate them. In February 2026, OpenAI reported an audit of 138 Verified tasks that one of its models did not solve consistently across 64 runs. Each audited task was independently reviewed by at least six experienced software engineers. OpenAI judged 59.4% of that selected 138-task subset to have material test-design or problem-description issues. Because the audit deliberately focused on model-hard tasks, 59.4% must not be presented as the defect rate of all 500 Verified tasks.^[11]

The same OpenAI report said that frontier models could reproduce exact reference fixes or problem-specific details for some public tasks, which it treated as evidence of training-data contamination. OpenAI concluded that Verified no longer provided meaningful signal for comparing frontier launches at the performance levels it studied and stopped reporting it for that purpose. That conclusion is narrower than saying the dataset has no research value: Verified may still support historical comparisons, system debugging, or studies conducted with explicit contamination controls.^[11]

The SWE-Bench+ and OpenAI studies examined different samples and questions. SWE-Bench+ started from successful patches for one model-and-agent configuration and looked for leakage or weak tests among those successes. OpenAI selected tasks that a later model failed inconsistently and looked for false negatives and contamination at a high-performance frontier. Their percentages should not be pooled, averaged, or treated as interchangeable estimates of a single benchmark-wide defect rate.^[10][11]

Test validity and contamination

SWE-bench tasks originate in public project history. Issue discussions may include implementation hints, and the accepted pull request, tests, release notes, and later explanations can become available on the web. A model trained after those materials were published may have encountered all or part of a task. High performance can therefore mix repository reasoning with remembered task-specific information. The degree of contamination is difficult to infer from a score alone.^[10][11]

Tests create a second validity problem. Project tests were written to validate a particular contribution, not necessarily to define every functionally acceptable solution to a standalone prompt. A hidden test may require a helper name, exact message, or extra behavior not stated in the issue. Conversely, a narrow test suite can overlook an incomplete implementation. The accepted human patch is useful for constructing an environment and tests, but difference from that patch is not itself evidence that a generated solution is wrong.^[6][10][11]

Static public benchmarks also invite optimization to a fixed set. Researchers can inspect failures, tune scaffolds, or add task-specific heuristics over repeated evaluation cycles. Such work can improve useful engineering methods, but it makes a public score less informative about generalization to unseen repositories. Fresh or held-out tasks are one response, provided their test quality is audited rather than assumed.

Later benchmarks and the 2026 reassessment

SWE-Bench Pro, released by Scale AI in 2025, was designed around longer tasks and a broader set of repositories. Its paper describes 1,865 tasks from 41 repositories: 731 public tasks from 11 repositories, 858 held-out tasks from 12 repositories, and 276 tasks from 18 commercial repositories. The creators used human augmentation and verification and reported reference solutions averaging 107.4 changed lines across 4.1 files.^[12]

Pro was initially promoted as an answer to contamination and saturation in Verified. In July 2026, however, OpenAI published an audit of Pro's 731-task public split. Its pipeline flagged 200 tasks, or 27.4%, as broken, while a human annotation campaign identified 249, or 34.1%. The reported failure modes included overly strict tests, underspecified or misleading prompts, and tests with insufficient coverage. OpenAI estimated that roughly 30% of the public split was broken and retracted its earlier recommendation to adopt Pro.^[13] This audit does not make every Pro task invalid, but it shows that greater task length and private or held-out code do not by themselves guarantee a sound evaluation.

The Pro audit first used model attempts, metadata, and failure traces to flag 286 potentially problematic public tasks. OpenAI then used investigator agents and human reviewers for deeper analysis; each task in the human campaign was reviewed by five experienced software engineers. Human reviewers were more likely than the agent pipeline to assign more than one defect category. These methods explain why the two reported broken-task counts differ and why neither should be described as an unqualified ground-truth count for all possible evaluation setups.^[13]

SWE-bench-Live is a separate effort aimed at freshness. Its 2025 paper described an initial release of 1,319 tasks from 93 repositories, based on issues created since 2024, with a dedicated Docker image for each task and an automated curation pipeline.^[14] Recency can reduce some exposure risks, but a live benchmark still needs checks for prompt clarity, test coverage, environment stability, and unintended solution disclosure.

Private-codebase evaluations

A different response to contamination is to move the tasks off the public internet altogether. Real-SWE, published by Specific Labs in September 2026, draws each task from a private production codebase licensed from an operating company, on the argument that neither that code nor its accepted solution is available to a pretraining crawl. Tasks are packaged in Harbor format, run in isolated sandboxes, and graded by verifiers that are injected at grading time and are either taken from or modeled on the codebase's own test suite. Each entry names a model together with the command-line agent it ran in, which Specific describes as evaluating model-and-harness combinations rather than models in isolation.^[15]

The published figures are narrow. Resolution rate is pass@1 averaged over eight independent rollouts per task, and the per-task results cover 10 tasks for each of the eight systems, 640 rollouts in total, with all models run at high reasoning effort. Specific presents those ten as a sample of a larger task set and does not give the benchmark's full size, although the eight leaderboard percentages are exactly the ten-task averages. Six of those 10 tasks were resolved on fewer than 15% of rollouts and one on none at all, so the ranking rests on a small number of items. Because the task set is private, no outside group can rerun it, inspect the verifiers, or audit prompt quality the way OpenAI audited Verified and Pro, which leaves the numbers dependent on the vendor's own reporting. Specific Labs also sells access to the underlying asset: its stated business is turning licensed company data into training and evaluation material for AI models.^[15][16] The harness column carries as much of the variation as the model column: the two entries that ran in Claude Code differ by 10 percentage points, and the two that ran in Codex CLI differ by 17.6.^[15] Real-SWE covers the construction and results in more detail.

Interpreting results

SWE-bench is useful because it asks systems to operate on real repositories and judges executable changes. Its outcome is more concrete than a subjective rating of generated code. It can reveal failures in code localization, repository navigation, patch construction, tool use, debugging, and regression avoidance. It can also support controlled studies of retrieval-augmented generation, agent interfaces, or context management.

The benchmark should not be read as a direct percentage of software-engineering jobs a system can perform. Its repository sample, issue distribution, hidden-test design, time horizon, and success criterion cover only part of software engineering. A model that resolves a task may still produce code that is insecure, inefficient, poorly documented, or hard to maintain outside the tested behavior. A failed task may instead reflect a broken environment or an unfair hidden test.

Automated grading versus maintainer review

A March 2026 research note from METR asked a narrower question than dataset quality: would a patch that satisfies the automated grader be accepted by the people who maintain the repository? Parker Whitfill, Cheryl Wu, Joel Becker, and Nate Rush recruited four active maintainers from three SWE-bench Verified repositories, two from scikit-learn, one from Sphinx, and one from pytest, covering 95 of the 500 Verified issues. The maintainers reviewed 296 AI-generated pull requests on GitHub as they would review real contributions, blind to whether a human or a model had written each one, and recorded an accept-or-request-changes decision plus a primary reason for rejection.^[17]

Maintainer review is subjective, so METR calibrated it against 47 original human patches that had actually been merged into main. Maintainers accepted about 68% of those, and METR reports every score as a percentage of that golden baseline. On that scale, roughly half of the test-passing patches produced by agents released between mid-2024 and mid-to-late 2025 would not have been merged. Across the five systems studied, Claude 3.5 Sonnet, Claude 3.7 Sonnet, Claude Opus 4, Claude Sonnet 4.5, and GPT-5, with patches drawn from Epoch AI's benchmarking hub, the automated grader scored an average of 24.2 percentage points higher than the maintainer merge decision (standard error 2.7). The rejection categories were core functionality failure, breaking other code, and code quality.^[17]

METR is explicit about what the result is not. Agents submitted a single patch with no opportunity to iterate on review feedback, so the gap is not evidence of a fixed capability ceiling; METR expects better prompting and elicitation to remove some of it. METR also treats its secondary finding, that maintainer merge rates improve about 9.6 percentage points per year more slowly than grader scores, as weak: it is significant only at the 10% level and it shrinks and loses significance when the sample is restricted to models that were state of the art at release. The claim METR does make is that reading a Verified percentage as the share of real issues an agent can resolve overstates how useful the agent is without further elicitation or human feedback.^[17]

An August 2025 METR post reached a similar conclusion from a different sample. Taking 18 issues from the stdlib-js and hypothesis repositories, David Rein found that a Claude 3.7 Sonnet agent passed the original authors' test cases on about 38% of attempts, but that none of the 15 submissions reviewed by hand were mergeable as they stood. Inadequate test coverage, missing documentation, linting and formatting problems, and general code-quality issues appeared in most of them, including in runs that passed the tests.^[18] Neither study is a reason to discard execution-based scoring, but both argue that a resolved rate is an upper bound on the share of work a maintainer would accept.

The safest interpretation is comparative and protocol-specific: compare systems on the same audited task set, with the same harness and clearly disclosed budgets, then inspect task-level failures and confidence intervals. For consequential claims, evaluators should manually audit a sample of both successes and failures, check for contamination, and report sensitivity to removing disputed tasks. A single aggregate score is evidence, not a complete capability assessment.

For benchmark design, the later audits suggest four separate quality gates: the prompt must specify the behavior that tests enforce; tests must accept reasonable alternative implementations; test coverage must reject materially incomplete fixes; and the environment must reproduce reliably. Freshness and hidden data address contamination, but they do not replace these validity checks. Conversely, a carefully audited static set can still be valuable even after it becomes too easy for ranking frontier systems, because it can support regression tests and controlled ablations.

See also

• AI agents
• International Conference on Learning Representations
• Multi-SWE-bench
• SWE-rebench

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024; arXiv:2310.06770. arxiv.org/...2310.06770
2. ^SWE-bench project. "SWE-bench Datasets." Official documentation. swebench.com/...datasets
3. ^1 ^2 ^3SWE-bench project. "Evaluation Guide." Official documentation. swebench.com/...evaluation
4. ^SWE-bench project. "Frequently Asked Questions." Official documentation. swebench.com/...faq
5. ^1 ^2SWE-bench project. "SWE-bench Lite." Official benchmark page. swebench.com/lite
6. ^1 ^2 ^3 ^4OpenAI. "Introducing SWE-bench Verified." August 13, 2024; updated February 24, 2025. openai.com/...introducing-swe-bench-verified
7. ^1 ^2John Yang et al. "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?" arXiv:2410.03859. arxiv.org/...2410.03859
8. ^1 ^2SWE-bench project. "SWE-bench Multilingual." Official benchmark page. swebench.com/multilingual
9. ^1 ^2SWE-bench project. "Submit to SWE-bench." Official submission documentation. swebench.com/submit
10. ^1 ^2 ^3 ^4Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. "SWE-Bench+: Enhanced Coding Benchmark for LLMs." arXiv:2410.06992. arxiv.org/...2410.06992
11. ^1 ^2 ^3 ^4 ^5OpenAI. "Why SWE-bench Verified no longer measures frontier coding capabilities." February 23, 2026. openai.com/...o-longer-evaluate-swe-bench-verified
12. ^Xiang Deng et al. "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?" Scale AI, 2025. labs.scale.com/...swe_bench_pro
13. ^1 ^2OpenAI. "Separating signal from noise in coding evaluations." July 8, 2026. openai.com/...signal-from-noise-coding-evaluations
14. ^Linghao Zhang et al. "SWE-bench Goes Live!" arXiv:2505.23419. arxiv.org/...2505.23419
15. ^1 ^2 ^3Snagnik Das, Siddhant Paliwal, and Janak Sunil. "Introducing Real-SWE." Specific Labs, September 2026. withspecific.com/...real-swe
16. ^Specific Labs. "Company data partnerships." withspecific.com/company-data
17. ^1 ^2 ^3Parker Whitfill, Cheryl Wu, Joel Becker, and Nate Rush. "Many SWE-bench-Passing PRs Would Not Be Merged into Main." METR, March 10, 2026. metr.org/...sing-prs-would-not-be-merged-into-main
18. ^David Rein. "Research Update: Algorithmic vs. Holistic Evaluation." METR, August 13, 2025. metr.org/...econciling-slowdown-with-time-horizons