# SWE-bench

> Source: https://aiwiki.ai/wiki/swe_bench
> Updated: 2026-07-29
> Fact-checked: 2026-07-29
> Categories: AI Benchmarks, Software Development
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "SWE-bench." aiwiki.ai, 29 Jul 2026. https://aiwiki.ai/wiki/swe_bench
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

SWE-bench is an execution-based [benchmark](https://aiwiki.ai/wiki/benchmark) for evaluating whether a language-model system can resolve real software issues. Each task provides a repository at a specified commit and the text of a resolved issue; the system must produce a code patch that passes tests for the requested fix without breaking selected existing behavior. The original benchmark contains 2,294 tasks drawn from 12 open-source Python repositories. Carlos E. Jimenez and six coauthors introduced it in 2023, and the paper appeared at ICLR 2024.[1] A SWE-bench score therefore measures an evaluated system - including its model, agent scaffold, tools, context-selection method, and evaluation setup - rather than a language model in isolation.

## Task construction

The original authors mined merged pull requests from 12 established [Python](https://aiwiki.ai/wiki/python) repositories. A candidate pull request had to resolve at least one [GitHub](https://aiwiki.ai/wiki/github) issue and change test files. The construction pipeline then compared tests before and after applying the pull request. At least one test had to change from failing to passing, while other selected tests had to keep passing. After filtering an initial collection of more than 90,000 pull requests, the process produced 2,294 benchmark instances.[1]

An instance identifies the repository, the base commit, the issue-derived problem statement, the reference patch, and a patch containing tests. It also records two test groups: `FAIL_TO_PASS`, for tests expected to fail before the fix and pass afterward, and `PASS_TO_PASS`, for regression tests expected to pass both before and after the fix. The reference patch and test patch are evaluator artifacts; they are not a standard solver's answer or prompt. The official dataset documentation also records fields such as an instance identifier, repository version, issue URL, pull-request URL, and creation date.[2]

This design makes the unit of evaluation larger than a short code-completion exercise. A task can require locating relevant code in a repository, understanding behavior across functions or files, editing the working tree, and returning an applicable diff. It does not, however, reproduce every part of professional software development: the benchmark normally ends when its selected tests pass, not after code review, deployment, monitoring, or long-term maintenance.[1]

### Original dataset profile

The original test set is unevenly distributed across its 12 repositories. Django contributes 850 of the 2,294 tasks, while Flask contributes 11. Repository-level differences can therefore materially affect an aggregate score.

| Repository | Tasks | Repository | Tasks |
| --- | ---: | --- | ---: |
| astropy | 95 | django | 850 |
| flask | 11 | matplotlib | 184 |
| pylint | 57 | pytest | 119 |
| requests | 44 | scikit-learn | 229 |
| seaborn | 22 | sphinx | 187 |
| sympy | 386 | xarray | 110 |

The paper reports an average issue length of 195.1 words and an average non-test codebase size of about 438,000 lines. Human reference patches edited an average of 32.8 lines, 1.7 files, and 3 functions. These are means with wide ranges: the largest reference patch edited 5,888 lines and 31 files. Such variation is one reason that a single resolution percentage conceals substantial differences in task type and effort.[1]

The repository mix also limits external validity. All 12 original projects are open-source Python packages, and several are scientific-computing or developer-tool projects. Results do not automatically transfer to private services, mobile applications, embedded systems, security-sensitive code, or projects written in other languages. Later variants expanded language and repository coverage, but they are separate evaluations rather than retroactive changes to the original set.[1]

## Evaluation protocol

For each prediction, the evaluation harness creates a containerized repository environment, applies the proposed patch, and runs the task's tests. A task is resolved only when the patch applies and all required `FAIL_TO_PASS` and `PASS_TO_PASS` tests pass. The principal aggregate metric is the percentage of evaluated instances resolved.[1][3]

The two test groups serve different purposes:

- `FAIL_TO_PASS` tests are intended to verify the new behavior associated with the issue.
- `PASS_TO_PASS` tests are intended to detect regressions in behavior that already worked.
- A patch that fixes the target test but breaks a selected regression test is not resolved.
- A malformed or non-applicable patch cannot be counted as resolved even if its intended edit appears plausible.

The official evaluation documentation specifies a prediction record containing `instance_id`, `model_name_or_path`, and `model_patch`. It also reports counts for submitted, completed, resolved, empty, and errored predictions. Consequently, a published result should state its denominator and how incomplete or errored instances were handled, not only a percentage.[4][3]

The tests are a programmatic oracle, not a proof that a patch is the only correct implementation. A correct alternative can be rejected if a hidden test enforces an unstated detail, while an incomplete patch can pass if coverage is weak. Environment configuration can also change outcomes. These limitations became a major subject of later audits.

### What the score combines

An issue-resolution run generally has at least three stages: locating relevant code, proposing edits, and validating or revising those edits. A system may expose the model to a fixed retrieval result, let an agent search the repository interactively, or allow the agent to run tests and inspect failures. These are materially different experimental conditions even when they use the same model checkpoint.

The standard resolved metric is binary at task level. It does not award partial credit for finding the right file, identifying the defect, writing a nearly correct patch, or passing some but not all required tests. That simplicity makes aggregation straightforward, but it also means that two systems with the same resolution rate can fail for very different reasons. Task-level logs are needed to distinguish localization failures, malformed patches, timeouts, environment failures, and semantically wrong solutions.

Repeated sampling adds another distinction. Pass@1 describes one attempt per task. Pass@k or a best-of-k procedure gives a system multiple chances and normally yields a higher probability of at least one passing patch. Neither statistic is inherently improper, but they answer different questions and have different compute costs. A comparison that omits the attempt count can be misleading.

## Core datasets and variants

Several datasets use the SWE-bench name but differ in task count, repository coverage, modality, or curation. Their scores are not interchangeable.

| Dataset | Scope | Size and split | Distinguishing feature |
| --- | --- | --- | --- |
| SWE-bench | Original benchmark | 2,294 test tasks from 12 Python repositories | Broad original issue-resolution set.[1] |
| SWE-bench Lite | Subset of the original | 300 test tasks plus 23 development tasks; 11 of the 12 repositories | Focuses on more self-contained functional bug fixes and removes several costly or externally dependent cases.[5] |
| SWE-bench Verified | Human-screened subset of the original | 500 test tasks | Filters for better-specified problems and more appropriate tests.[6] |
| SWE-bench Multimodal | Separate visual-software dataset | 619 tasks in the paper body: 102 development and 517 test tasks, from 17 JavaScript repositories | Includes visual information in problem statements or tests. The paper's abstract says 617, while its construction section and split totals say 619.[7] |
| SWE-bench Multilingual | Separate multilingual dataset | 300 tasks from 42 repositories in 9 programming languages | Extends repository-level issue resolution beyond Python.[8] |

The SWE-bench Lite test set was sampled after excluding, among other cases, tasks with images, external hyperlinks, edits to more than one file, large reference patches, file creation or deletion, and certain error-message tests. Those choices reduce evaluation cost but also change the task distribution. Lite is therefore not merely a cheaper run of the full set.[5]

OpenAI and the SWE-bench authors created [SWE-bench Verified](https://aiwiki.ai/wiki/swe-bench_verified) after a 2024 annotation campaign. Ninety-three developers experienced with Python reviewed 1,699 randomly selected original tasks, with three independent labels per task. OpenAI reported that 38.3% of reviewed samples were flagged for underspecified problem statements and 61.1% for tests that could unfairly reject valid solutions; 68.3% were removed for these or other issues. The released subset contains 500 tasks, including 196 estimated to require less than 15 minutes and 45 estimated to require more than one hour for an experienced developer.[6]

For its release analysis, OpenAI evaluated the dated `gpt-4o-2024-05-13` model with several open-source scaffolds. It reported 33.2% resolved for the best of those configurations, Agentless, using a single seed and the authors' closest-documented or default settings. That number is useful as a release-era baseline, but the source itself cautions that it may differ from official leaderboard submissions.[6]

The published [SWE-bench Multimodal](https://aiwiki.ai/wiki/swe_bench_multimodal) paper has an internal count discrepancy. Its abstract reports 617 tasks, but the dataset-construction section says that annotators removed 24 impossible tasks to leave 619, divided into 102 development and 517 test tasks. The latter split sums to 619. Reporting the split explicitly avoids silently choosing between the paper's conflicting totals.[7]

[SWE-bench Multilingual](https://aiwiki.ai/wiki/swe_bench_multilingual) contains 300 manually verified tasks from 42 repositories. It covers C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, and Rust while retaining the issue-to-patch evaluation pattern.[8] These variants test different distributions, so a result should always name the exact dataset and split.

## Original baselines

The original paper evaluated then-current language models using two ways of selecting repository context. A BM25 retriever selected files based on lexical relevance to the issue text. An "oracle" condition instead supplied files changed by the human reference patch. The oracle condition was an analysis tool, not a realistic deployment setting, because it used knowledge of which files the accepted solution edited.[1]

With BM25 retrieval, the paper reported a 1.96% resolution rate for Claude 2 on the full 2,294-task set. This result established that the original task was difficult for the evaluated 2023-era setup. It is a historical baseline, not a current leaderboard value and not directly comparable with later results on Lite, Verified, or other variants. Even within one dataset, changing file retrieval, prompting, tools, context limits, patch format, or the number of attempts changes the evaluated system.[1]

The retrieval experiments illustrate this dependency. At a 27,000-token limit, BM25 retrieved at least one file changed by the reference patch for about half of the instances, but it retrieved none of those files for nearly the other half. Supplying more retrieved context did not consistently improve resolution. The paper attributed part of this behavior to difficulty localizing the relevant code amid distracting context.[1]

The authors also trained SWE-Llama, based on 7-billion- and 13-billion-parameter Code Llama models, as a repository-editing baseline. They collected about 19,000 issue and pull-request pairs from 37 repositories that did not overlap the test repositories. After excluding training sequences longer than 30,000 tokens, about 10,000 instances remained for fine-tuning. These models were research baselines associated with the original paper, not additional SWE-bench test sets.[1]

## Leaderboards and reproducibility

The official leaderboards accept results for several SWE-bench variants. Their submission page points to a public experiments repository that stores official submissions and directs evaluators to the current harness or cloud tooling. Multimodal evaluation has a separate submission path because its test data are not distributed in the same way as the ordinary text-only sets.[9]

A leaderboard row should be interpreted as a configuration, not as a timeless property of a named model. At minimum, a reproducible report should identify:

- the exact dataset, split, and dataset revision;
- the model identifier or dated API version;
- the agent scaffold and its version;
- allowed tools, repository access, retrieval method, and image input where applicable;
- the evaluation-harness version and environment;
- the number of attempts per task and whether the statistic is pass@1, pass@k, or another estimator;
- token, time, and monetary budgets;
- the number of submitted, completed, errored, and resolved tasks; and
- prediction files or trajectories when release is permitted.

Comparisons that change several of these variables at once cannot isolate an improvement in the underlying [large language model](https://aiwiki.ai/wiki/large_language_model). Likewise, the highest live leaderboard entry can change after an article is written. A durable encyclopedia article is better served by explaining the protocol and linking to the maintained leaderboard than by copying a fast-aging ranking table.

Containerization improves consistency but does not remove every source of drift. A container build can depend on package registries, archived dependencies, base images, CPU architecture, or network access. Harness updates can repair task specifications or environment scripts. For reproducibility, a result should preserve the predictions, task-level logs, container or image identifiers, and exact harness revision in addition to the headline score.[3][9]

Confidence also depends on sample size and task dependence. A 300-task subset has wider sampling uncertainty than a 2,294-task set, and tasks from the same repository are not necessarily independent. A repository-weighted analysis may answer a different question from a task-weighted average dominated by Django. Reporting per-repository results and uncertainty can reveal whether an apparent gain is broad or concentrated in a few projects.

## Data-quality studies

The SWE-Bench+ preprint examined 251 patches that SWE-Agent with GPT-4 had successfully submitted on the full benchmark and that the authors confirmed as passing the recorded tests. Among this conditional sample of successful patches, the researchers classified 82 patches, or 32.67%, as cases where the issue or comments disclosed the solution. They classified another 78, or 31.08%, as incorrect, incomplete, or misplaced fixes that passed weak tests. These percentages describe the reviewed successful-patch sample; they are not estimates that the same fractions of all 2,294 tasks are defective.[10]

Verified was designed to reduce underspecification and test mismatch, but later evidence showed that manual filtering did not eliminate them. In February 2026, OpenAI reported an audit of 138 Verified tasks that one of its models did not solve consistently across 64 runs. Each audited task was independently reviewed by at least six experienced software engineers. OpenAI judged 59.4% of that selected 138-task subset to have material test-design or problem-description issues. Because the audit deliberately focused on model-hard tasks, 59.4% must not be presented as the defect rate of all 500 Verified tasks.[11]

The same OpenAI report said that frontier models could reproduce exact reference fixes or problem-specific details for some public tasks, which it treated as evidence of training-data contamination. OpenAI concluded that Verified no longer provided meaningful signal for comparing frontier launches at the performance levels it studied and stopped reporting it for that purpose. That conclusion is narrower than saying the dataset has no research value: Verified may still support historical comparisons, system debugging, or studies conducted with explicit contamination controls.[11]

The SWE-Bench+ and OpenAI studies examined different samples and questions. SWE-Bench+ started from successful patches for one model-and-agent configuration and looked for leakage or weak tests among those successes. OpenAI selected tasks that a later model failed inconsistently and looked for false negatives and contamination at a high-performance frontier. Their percentages should not be pooled, averaged, or treated as interchangeable estimates of a single benchmark-wide defect rate.[10][11]

## Test validity and contamination

SWE-bench tasks originate in public project history. Issue discussions may include implementation hints, and the accepted pull request, tests, release notes, and later explanations can become available on the web. A model trained after those materials were published may have encountered all or part of a task. High performance can therefore mix repository reasoning with remembered task-specific information. The degree of contamination is difficult to infer from a score alone.[10][11]

Tests create a second validity problem. Project tests were written to validate a particular contribution, not necessarily to define every functionally acceptable solution to a standalone prompt. A hidden test may require a helper name, exact message, or extra behavior not stated in the issue. Conversely, a narrow test suite can overlook an incomplete implementation. The accepted human patch is useful for constructing an environment and tests, but difference from that patch is not itself evidence that a generated solution is wrong.[6][10][11]

Static public benchmarks also invite optimization to a fixed set. Researchers can inspect failures, tune scaffolds, or add task-specific heuristics over repeated evaluation cycles. Such work can improve useful engineering methods, but it makes a public score less informative about generalization to unseen repositories. Fresh or held-out tasks are one response, provided their test quality is audited rather than assumed.

## Later benchmarks and the 2026 reassessment

[SWE-Bench Pro](https://aiwiki.ai/wiki/swe_bench_pro), released by Scale AI in 2025, was designed around longer tasks and a broader set of repositories. Its paper describes 1,865 tasks from 41 repositories: 731 public tasks from 11 repositories, 858 held-out tasks from 12 repositories, and 276 tasks from 18 commercial repositories. The creators used human augmentation and verification and reported reference solutions averaging 107.4 changed lines across 4.1 files.[12]

Pro was initially promoted as an answer to contamination and saturation in Verified. In July 2026, however, OpenAI published an audit of Pro's 731-task public split. Its pipeline flagged 200 tasks, or 27.4%, as broken, while a human annotation campaign identified 249, or 34.1%. The reported failure modes included overly strict tests, underspecified or misleading prompts, and tests with insufficient coverage. OpenAI estimated that roughly 30% of the public split was broken and retracted its earlier recommendation to adopt Pro.[13] This audit does not make every Pro task invalid, but it shows that greater task length and private or held-out code do not by themselves guarantee a sound evaluation.

The Pro audit first used model attempts, metadata, and failure traces to flag 286 potentially problematic public tasks. OpenAI then used investigator agents and human reviewers for deeper analysis; each task in the human campaign was reviewed by five experienced software engineers. Human reviewers were more likely than the agent pipeline to assign more than one defect category. These methods explain why the two reported broken-task counts differ and why neither should be described as an unqualified ground-truth count for all possible evaluation setups.[13]

SWE-bench-Live is a separate effort aimed at freshness. Its 2025 paper described an initial release of 1,319 tasks from 93 repositories, based on issues created since 2024, with a dedicated Docker image for each task and an automated curation pipeline.[14] Recency can reduce some exposure risks, but a live benchmark still needs checks for prompt clarity, test coverage, environment stability, and unintended solution disclosure.

## Interpreting results

SWE-bench is useful because it asks systems to operate on real repositories and judges executable changes. Its outcome is more concrete than a subjective rating of generated code. It can reveal failures in code localization, repository navigation, patch construction, tool use, debugging, and regression avoidance. It can also support controlled studies of [retrieval-augmented generation](https://aiwiki.ai/wiki/retrieval_augmented_generation), agent interfaces, or context management.

The benchmark should not be read as a direct percentage of software-engineering jobs a system can perform. Its repository sample, issue distribution, hidden-test design, time horizon, and success criterion cover only part of software engineering. A model that resolves a task may still produce code that is insecure, inefficient, poorly documented, or hard to maintain outside the tested behavior. A failed task may instead reflect a broken environment or an unfair hidden test.

The safest interpretation is comparative and protocol-specific: compare systems on the same audited task set, with the same harness and clearly disclosed budgets, then inspect task-level failures and confidence intervals. For consequential claims, evaluators should manually audit a sample of both successes and failures, check for contamination, and report sensitivity to removing disputed tasks. A single aggregate score is evidence, not a complete capability assessment.

For benchmark design, the later audits suggest four separate quality gates: the prompt must specify the behavior that tests enforce; tests must accept reasonable alternative implementations; test coverage must reject materially incomplete fixes; and the environment must reproduce reliably. Freshness and hidden data address contamination, but they do not replace these validity checks. Conversely, a carefully audited static set can still be valuable even after it becomes too easy for ranking frontier systems, because it can support regression tests and controlled ablations.

## See also

- [AI agents](https://aiwiki.ai/wiki/ai_agents)
- [International Conference on Learning Representations](https://aiwiki.ai/wiki/iclr)
- [Multi-SWE-bench](https://aiwiki.ai/wiki/multi_swe_bench)
- [SWE-rebench](https://aiwiki.ai/wiki/swe_rebench)

## References

1. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024; arXiv:2310.06770. https://arxiv.org/abs/2310.06770
2. SWE-bench project. "SWE-bench Datasets." Official documentation. https://www.swebench.com/SWE-bench/guides/datasets/
3. SWE-bench project. "Evaluation Guide." Official documentation. https://www.swebench.com/SWE-bench/guides/evaluation/
4. SWE-bench project. "Frequently Asked Questions." Official documentation. https://www.swebench.com/SWE-bench/faq/
5. SWE-bench project. "SWE-bench Lite." Official benchmark page. https://www.swebench.com/lite.html
6. OpenAI. "Introducing SWE-bench Verified." August 13, 2024; updated February 24, 2025. https://openai.com/index/introducing-swe-bench-verified/
7. John Yang et al. "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?" arXiv:2410.03859. https://arxiv.org/abs/2410.03859
8. SWE-bench project. "SWE-bench Multilingual." Official benchmark page. https://www.swebench.com/multilingual.html
9. SWE-bench project. "Submit to SWE-bench." Official submission documentation. https://www.swebench.com/submit.html
10. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. "SWE-Bench+: Enhanced Coding Benchmark for LLMs." arXiv:2410.06992. https://arxiv.org/abs/2410.06992
11. OpenAI. "Why SWE-bench Verified no longer measures frontier coding capabilities." February 23, 2026. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/
12. Xiang Deng et al. "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?" Scale AI, 2025. https://labs.scale.com/papers/swe_bench_pro
13. OpenAI. "Separating signal from noise in coding evaluations." July 8, 2026. https://openai.com/index/separating-signal-from-noise-coding-evaluations/
14. Linghao Zhang et al. "SWE-bench Goes Live!" arXiv:2505.23419. https://arxiv.org/abs/2505.23419