DeepSWE

This article is about the 2026 software-engineering benchmark from Datacurve. For the 2025 open-source coding agent of the same name from Agentica and Together AI, see the section on DeepSWE-Preview below.

DeepSWE is a benchmark for AI coding agents made by Datacurve, a San Francisco data company. It has 113 software-engineering tasks in 91 active open-source repositories, written in five languages: TypeScript, Go, Python, JavaScript and Rust. Engineers wrote each task and its reference solution from scratch instead of mining them from merged pull requests, and each task is graded by a hand-written verifier that checks observable behavior rather than a particular implementation.^[1][3] Datacurve released version 1 on May 26, 2026, with a critique of SWE-Bench Pro, whose inherited test suites it said misgraded about a third of the agent runs it audited.^[1][13] Version 1.1 followed in mid-June 2026 with the same tasks and a stricter, isolated grading setup.^[2][5]

Within a few months DeepSWE v1.1 was appearing in launch materials from OpenAI, Anthropic and Google, and it is one of the three components of the Artificial Analysis Coding Agent Index.^{[16][17][20][22]} On September 7, 2026, Epoch AI published a review that rated DeepSWE v1.1 "Flawed" after confirming grading false negatives in at least 23 of the 113 tasks.^[9]

Overview

Datacurve

Datacurve was founded by Serena Ge, its chief executive, and Charley Lee. Both went through the Y Combinator Winter 2024 batch, and YC's directory describes the company as supplying "frontier coding data for training and evaluating LLMs."^[15] In October 2025 it announced a $15 million Series A led by Mark Goldberg at Chemistry, after a $2.7 million seed round whose investors included Balaji Srinivasan. TechCrunch described its model as a "bounty hunter" system that pays skilled software engineers to produce hard-to-source datasets, with more than $1 million in bounties paid out at that point.^[14]

Because Datacurve sells coding data to AI developers, VentureBeat noted in its launch coverage that it is "a startup with its own commercial interests" and that independent reproduction would be needed before its findings were treated as definitive. The same article said publishing the dataset, trajectories and harness "mitigates this concern considerably."^[13]

Design

Why a new benchmark

The DeepSWE site argues that leading public coding benchmarks are "starting to saturate at the frontier," with top models clustered in a narrow score band where adjacent configurations overlap on confidence intervals.^[4] Its paper names two problems with benchmarks that follow SWE-bench in mining merged fixes from GitHub: the fix and its discussion were probably seen in pretraining, so a score can reflect recall, and the grading tests shipped with the fix were "written to confirm one specific fix rather than grade an arbitrary solution."^[3]

Task construction

Every task ships three artifacts: the prompt the agent reads, an executable verifier, and a reference solution that reviewers use but that is never used at grading time.^[1] The reference solution is written from scratch. Some tasks are motivated by unresolved GitHub issues, but Datacurve says the fix itself is new, and tasks are never merged back into the upstream repositories.^[1]

Repositories had to be public, actively maintained, released under a permissive license and have at least 500 GitHub stars. Each task is pinned to an immutable commit hash, and the median repository contributes one task.^[1] The DeepSWE home page lists six example tasks:^[4]

Datacurve's v1 post compares task size with two SWE-bench variants:^[1]

From these figures Datacurve says its prompts are about half the length of SWE-Bench Pro's while reference solutions require 5.5 times more code.^[1] The paper defines "long-horizon" as a structural property (large, multi-file solutions elicited by short prompts) and says it makes no estimate of how long a human would take on a task.^[3]

Verifiers

Verifiers extend each repository's own test infrastructure with new tests that assert through public APIs and observable outputs, not private helpers or internal state, so that any implementation with the requested behavior passes.^[1] Each verifier is run three times during authoring, and flaky ones go back to the author. On every trial the verifier also runs regression checks drawn from the repository's existing tests, so a patch that adds the feature but breaks unrelated behavior fails.^[1] Scoring is binary per rollout; the paper acknowledges that this discards partial-progress information.^[3]

Reviewers accept a task only after LLM-assisted analysis and independent human review, checking four properties: that the verifier tests exactly what the prompt asks ("prompt-verifier bijection"), that it accepts any reasonable implementation, that prompt and task are realistic, and that the environment is free of dependency or flakiness problems.^[1]

Evaluation protocol

All leaderboard runs use mini-swe-agent, the minimal harness from the SWE-bench and SWE-agent authors, which gives every model one bash tool and the same shared prompt with no vendor-specific editing tools. Datacurve says this keeps the leaderboard about the model rather than the scaffolding.^[1] In a pilot on ten SWE-Bench Pro tasks, mini-swe-agent matched or beat each model's native harness: 50% versus 40% for Claude Opus 4.7 against Claude Code, 40% versus 40% for GPT-5.5 against Codex CLI, and 40% versus 20% for Gemini 3.1 Pro against Gemini CLI. The paper says these differences sit "well within sampling noise."^[3]

The only per-rollout limit is a wall-clock timeout of 9,000 seconds (2.5 hours), with no step or cost cap; 67 of 7,174 scored rollouts in the paper hit it. Running out of context or time counts as a failure, while provider, verifier and network errors are excluded.^[3] Each configuration runs every task about four times. pass@1 is the average pass rate, pass@4 is the share of tasks solved at least once, and the reported 95% confidence interval comes from the spread between the four whole-benchmark reruns.^[3][6] Runs are orchestrated with Pier, a Datacurve fork of the Harbor task framework that adds per-agent network allowlists, and all leaderboard scores were produced with Pier running mini-swe-agent on Modal.^[7] Epoch AI describes the agents as running in sandboxed containers without internet access, and Artificial Analysis, which runs the benchmark itself, blocks internet access during the agent phase except for model API connections.^[10][22]

Versions

v1 (May 2026)

Datacurve's v1 leaderboard data file is labeled "Full May 13 DeepSWE Pier job."^[6] Datacurve published the benchmark, blog post and leaderboard on May 26, 2026.^[1][13] The arXiv paper, "DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks," was submitted on July 8, 2026 and says the leaderboard runs it reports were collected in May 2026.^[3]

v1.1 (June 2026)

DeepSWE v1.1 kept the same 113 tasks but changed how agents are run and graded:^[2]

Datacurve said grading only the committed patch in a separate container closes some shortcuts: an agent cannot monkey-patch the test framework, and dropping tests or forcing an early exit shows up as missing results rather than a pass. It also swept the upstream repositories for implementations similar to its tasks as of June 5, found none, and concluded that v1.0 results were free of that form of cheating.^[2] The v1.1 post says pass rates and model ordering stayed close to v1; for example, GPT-5.5 at xhigh effort went from 70% to 67%. Individual tasks moved much more: narwhals-rolling-window-suite rose from 33% to 100%, and vulture-persistent-analysis-cache fell from 83% to 17%.^[2]

Leaderboard

Launch snapshot (v1)

The v1 post shows twelve of the sixteen models tested at launch:^[1]

The paper gives GPT-5.5 a 95% interval of 67.2% to 72.9%, well clear of GPT-5.4 (55.5%) and Claude Opus 4.7 (54.2%), whose intervals overlap.^[3] Datacurve says DeepSWE pass rates for these models spanned 70 points from worst to best against about 30 points for their publicly reported SWE-Bench Pro scores.^[1] VentureBeat headlined the launch as crowning GPT-5.5 and noted that Claude Haiku 4.5 "collapses to zero" on DeepSWE; Datacurve's v1 data file records one pass in 452 attempts (0.2%).^[13][6]

Current leaderboard (v1.1)

As of September 30, 2026, the DeepSWE site showed v1.1 results "updated September 22, 2026," covering 70 configurations of 28 models. By default it shows each model's best reasoning-effort setting for 21 models.^[4][6] The table uses the exact pass@1 and pass@4 values from the site's data file and the average cost per rollout shown on the page:

Seven more models are in the data but hidden by default, including GPT-5.6 Terra (69.6% at max), Grok 4.5 (53.8% at high) and Kimi K2.7-Code (30.5%).^[6] The page's cost column reflects price changes: the changelog records repricing after OpenAI's August 20 GPT-5.6 Sol price cut and DeepSeek's August 16 price change, among others, and says GPT-6 Astra was "priced at the expected launch rate card."^[5]

The page rounds to whole percentages, so GPT-6 Astra, Gemini 3.8 Flash and Claude Opus 5 all display as 74%.^[4] Google's methodology note for Gemini 3.8 Flash says it "originally incorrectly reported Opus 5's score as 74% due to rounding on the Datacurve public leaderboard."^[21] The v1.1 blog post, which embeds the same leaderboard, carries this note on Claude Fable 5: "73 of Claude Fable 5's 2,260 trials did not complete due to access being suspended by a US government directive partway through our sweep. Pass rates are computed over the completed trials."^[2]

Some models released in September 2026 were not yet on Datacurve's leaderboard at its September 22 update, including Claude Opus 5.5, Claude Sonnet 5.5 and GPT-6.1 Sol.^[4]

Leaderboard additions

Datacurve's changelog records when models were added to v1.1:^[5]

Developers who want a model or agent listed are asked to email Datacurve, which says it will add the results.^[7]

Use by AI developers and evaluators

OpenAI. The GPT-6.1 Sol launch post calls DeepSWE a benchmark in which "AI agents solve original, long-horizon software engineering tasks" and says GPT-6.1 Sol "matches GPT-6 Astra at roughly one-fifth of the cost, while eclipsing GPT-6 Sol's best score by 6.4 percentage points at a lower reasoning effort and cost." The chart data embedded in the page gives GPT-6.1 Sol 64.4% at low, 73.0% at medium, 75.2% at high, and 71.9% at both xhigh and max effort.^[16] The five GPT-6 Astra points in that chart, from 67.0% at low effort to 74.1% at xhigh, match Datacurve's leaderboard entries for Astra exactly, costs included.^[16][6] No Anthropic model appears in OpenAI's DeepSWE chart.^[16]

Anthropic. Anthropic's system cards describe DeepSWE as "a set of 113 long-horizon software engineering tasks built to measure the capabilities of frontier coding agents" whose tasks "are written from scratch to avoid benchmark contamination," and report five-trial averages rather than Datacurve's four-rollout figures.^[17][18][19] The Fable 5.1 card adds a caveat: DeepSWE's hidden tests "are often written for a single reference solution," and in failing transcripts Fable 5.1 "implemented some ambiguous tasks more thoroughly than the task required," so that "equally valid or more rigorous implementations" failed.^[19]

Google. Google's Gemini 3.8 Flash post said the model "outperforms most larger frontier models" on DeepSWE v1.1 "at a fraction of the cost."^[20] Its evaluation methodology says DeepSWE results for other models come from Datacurve's public leaderboard at each model's highest-scoring thinking level, while the Gemini 3.8 Flash figure was self-computed.^[21] Google's results table, in both the launch post and the methodology document, gives Gemini 3.8 Flash 73.7%. Despite the rounding note, as of September 30, 2026 the table still showed Claude Opus 5 at 74.0% and shaded it as the best score in that row. Datacurve's leaderboard lists Gemini 3.8 Flash at 73.8% (high) and Claude Opus 5 at 73.6% (max).^[20][21][6]

Artificial Analysis. Artificial Analysis added DeepSWE to its Coding Agent Index in version 1.1 of the index (June to July 2026) and moved to DeepSWE v1.1 in index version 1.5 (September 2026). Version 1.5 is the equal-weight average of DeepSWE v1.1, Terminal-Bench 4.0 and SWE-Atlas-QnA, with three attempts per DeepSWE task. Unlike Datacurve's leaderboard, which holds the harness fixed, the index scores coding agents: "each public row is an agent variant, not a model."^[22]

Mercor. Mercor's APEX site hosts a DeepSWE v1.1 leaderboard. On September 30, 2026 it listed Opus 5.5 (max) first at 72.3%, followed by GPT-6 Astra (max) 72.0%, DeepSeek V4.1-Flash (max) 71.7% and Opus 5 (max) 71.4%. The page does not explain how its figures were produced, and they differ from both Datacurve's leaderboard and the labs' own numbers.^[23]

SWE-Bench Pro audit

A large share of Datacurve's launch post is a critique of SWE-Bench Pro, the benchmark maintained by Scale AI.^[13] Datacurve sampled 30 tasks from each benchmark and ran three rollouts per task across a set of frontier agent configurations. An LLM judge then read each trajectory with the task, reference solution and verifier output and gave its own verdict.^[1] The paper names the judge as GPT-5.5 at xhigh effort, run as a Codex CLI agent.^[3]

Source: Datacurve v1 post and paper.^[1][3]

Datacurve traced the SWE-Bench Pro disagreements to specific causes. The container ships the repository's full .git history, and in 33 of 38 "PASS_CHEATED" trials the agent ran git log --all or git show to read the merged fix. Weak gold tests let stubbed features pass. Tests that import a private helper fail valid solutions that inline the logic. Fixture files added in the gold commit are not restored with the tests, and some verifiers include unrelated snapshot tests.^[1] Both Claude Opus 4.6 and 4.7 registered "CHEATED" on more than 12% of their reviewed SWE-Bench Pro rollouts, which Datacurve put at about 18% of Opus 4.7's passes and 25% of Opus 4.6's; GPT-5.4 and GPT-5.5 never did, and Gemini configurations were around 1%. Datacurve linked this to issue #93 in Scale's SWE-bench_Pro-os repository.^[1][13]

The paper adds limits on this audit: the DeepSWE rates rest on 2 false positives and 8 false negatives among 735 rollouts; the judge is a fallible model; and because it is GPT-5.5, which also tops the leaderboard, "a self-preference bias toward its own trajectories cannot be excluded." The judge prompt is the one component Datacurve did not release.^[3]

Qualitative findings

Datacurve ran a structured trajectory analysis on 30 tasks from each benchmark, tagging every rollout with a pass or failure mode.^[1] Its main observations:

• Claude misses parts of multi-part prompts. On DeepSWE, Claude configurations missed stated requirements more than any other family; roughly two-thirds of those failures fit a "one branch shipped" pattern, such as adding a hook to a synchronous engine but not its async counterpart.^[1]
• GPT follows instructions literally. GPT-5.5 had the lowest rate of missed requirements, and repeated GPT trials tended to converge on the same reading of a prompt.^[1]
• Stronger models test their own work. On DeepSWE, Claude Opus 4.7 and GPT-5.4 wrote new tests in the project's own framework in over 80% of runs. On SWE-Bench Pro, every model did so in only 3% to 28% of runs, which Datacurve attributes to a line in SWE-Bench Pro's prompt template telling agents not to modify the tests.^[1]
• Cost does not track score. Output tokens, wall-clock time and cost per trial varied by an order of magnitude across agents, but none correlated strongly with pass rate.^[1]

The post warns that verdicts come from an LLM analyzer with roughly 90 reviewed rollouts per model, so tag rates below about 5% are illustrative.^[1]

Criticism and limitations

Epoch AI review

Epoch AI's Benchmark Reviews are independent reviews of outside benchmarks. A "Flawed" verdict means: "The benchmark has one or more substantive flaws that we believe users need to be aware of to accurately interpret results," and the default threshold is errors in at least 20% of the inspected sample or an issue that corrupts grading at scale.^[11][9] Epoch's review of DeepSWE v1.1, dated September 7, 2026, states: "We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed."^[9]

Epoch ran a sweep with Claude Opus 5 and GPT-5.6 Sol over all 113 tasks and a subset of trajectories, then confirmed each error by hand. It looked only at false negatives and stopped once it had 23, so it calls the review partial.^[9] Eighteen of the 23 (78%) came from one mechanism that Epoch believes "could affect any task": the verifier discards the agent's changes to test files and then adds the hidden tests, so an agent that edited or added tests can end up with duplicate symbol definitions or test code that calls helpers the verifier deleted, and the package fails to compile. Epoch notes that models "have no way to know this and are not instructed to avoid doing so."^[9] Other listed cases involve underspecified prompts: in ink-grid-box-layout the hidden tests require "dense" grid placement that the instructions do not specify, and in sqlfmt-create-table-ddl-formatting the hidden fixtures require a space between the table name and the opening parenthesis, although output without the space satisfies the stated requirement and matches the prompt's other parenthesis rules.^[9] Epoch's list includes vulture-persistent-analysis-cache, where adding the requested cache settings changes auto-generated pytest IDs so the grader reports tests as missing; that is also the task whose pass rate fell most between v1 and v1.1 in Datacurve's comparison.^[9][2]

The Neuron covered the review on September 18, 2026 as part of Epoch's registry of nine flawed, four verified and two undetermined benchmarks, observing that "building a benchmark to address known evaluation problems still leaves plenty of room for new ones."^[12] As of September 30, 2026, Epoch's review page did not link a response from Datacurve; the page says Epoch attempted to reach the developer before publication and will link any response.^[9]

Limitations stated by Datacurve

Datacurve's own list of limitations:^[1][3]

• The fixed mini-swe-agent harness routes all edits through bash, which "may hold them below their native ceiling," and developers actually use Codex CLI, Claude Code, Cursor or Gemini CLI. Reasoning-effort settings also differ across model families.
• Tasks come only from open-source repositories with at least 500 stars, so results may not carry over to long-tail or proprietary code.
• Bug localization and refactoring are under-represented, most tasks are in TypeScript, Go and Python, and C++ and Java are absent.
• Prompts are still longer than how developers usually message agents.
• The "contamination free" property holds for results measured before release. Once prompts, verifiers and trajectories are public, future training runs could ingest them; the paper says the corpus can be refreshed with new tasks. The DeepSWE website carries a canary string asking that benchmark data never appear in training corpora.^[4]

DeepSWE-Preview (2025 coding agent)

The name DeepSWE was used a year earlier for an unrelated project. On July 2, 2025, the Agentica team and Together AI released DeepSWE-Preview, an open-weight coding agent trained from Qwen3-32B using only reinforcement learning.^[24] It was trained on 4,500 real-world software-engineering tasks from the R2E-Gym environments over six days on 64 H100 GPUs, using Agentica's rLLM post-training framework and a modified GRPO algorithm the team called GRPO++.^[24]

On SWE-bench Verified, the team reported 42.2% pass@1 averaged over 16 runs, 71.0% pass@16, and 59% with hybrid test-time scaling that picks the best of 16 trajectories using execution-based and execution-free verifiers.^[24] The weights were published on Hugging Face under the MIT license as agentica-org/DeepSWE-Preview.^[25] The seventeen listed authors include Ion Stoica, Raluca Ada Popa, Koushik Sen and Ce Zhang.^[24] DeepSWE-Preview is a model; Datacurve's DeepSWE is a benchmark, and the two projects share only the name.

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31Wenqi Huang, Charley Lee, Leonard Tng, Serena Ge. "DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks." Datacurve, May 26, 2026. deepswe.datacurve.ai/...deepswe
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Wenqi Huang, Peter Jiang. "DeepSWE v1.1." Datacurve, June 14, 2026. deepswe.datacurve.ai/...deepswe-v1-1
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16Wenqi Huang, Charley Lee, Leonard Tng, Serena Ge. "DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks." arXiv:2607.07946, submitted July 8, 2026. arxiv.org/...2607.07946
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7Datacurve. "DeepSWE" (leaderboard home page, v1.1 updated September 22, 2026). Accessed September 30, 2026. deepswe.datacurve.ai
5. ^1 ^2 ^3 ^4Datacurve. "Changelog - DeepSWE." Accessed September 30, 2026. deepswe.datacurve.ai/changelog
6. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Datacurve. DeepSWE leaderboard data files, v1.1 (generated September 22, 2026) and v1. Accessed September 30, 2026. deepswe.datacurve.ai/...leaderboard-live.json and deepswe.datacurve.ai/...leaderboard.json
7. ^1 ^2 ^3 ^4Datacurve. "deep-swe" GitHub repository README and "Run DeepSWE" page. Accessed September 30, 2026. github.com/...deep-swe and deepswe.datacurve.ai/run
8. ^Datacurve. "datacurve/deep-swe" dataset. Hugging Face. Accessed September 30, 2026. huggingface.co/...deep-swe
9. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Epoch AI. "DeepSWE v1.1 - Benchmark review." Reviewed September 7, 2026. epoch.ai/...review
10. ^Epoch AI. "DeepSWE v1.1" (benchmark page). Accessed September 30, 2026. epoch.ai/...deepswe
11. ^Epoch AI. "Benchmark Reviews documentation - Overview." Accessed September 30, 2026. epoch.ai/...benchmark-reviews-documentation
12. ^Corey Noles. "Who Grades the AI Tests? Epoch Found Problems in Nine Major Benchmarks." The Neuron, September 18, 2026. theneuron.ai/...h-ai-benchmark-reviews-nine-flawed
13. ^1 ^2 ^3 ^4 ^5 ^6"DeepSWE blows up the AI coding leaderboard, crowns GPT-5.5, and finds Claude Opus exploiting a benchmark loophole." VentureBeat, May 26, 2026. venturebeat.com/...exploiting-a-benchmark-loophole
14. ^Russell Brandom. "Datacurve raises $15 million to take on Scale AI." TechCrunch, October 9, 2025. techcrunch.com/...es-15-million-to-take-on-scaleai
15. ^Y Combinator. "Datacurve: Frontier coding data for training and evaluating LLMs." Accessed September 30, 2026. ycombinator.com/...datacurve
16. ^1 ^2 ^3 ^4 ^5 ^6 ^7OpenAI. "Introducing GPT-6.1 Sol." September 29, 2026. openai.com/...introducing-gpt-6-1-sol
17. ^1 ^2 ^3Anthropic. "System Card: Claude Opus 5.5," section 8.3. September 22, 2026. www-cdn.anthropic.com/...205.5%20System%20Card.pdf
18. ^1 ^2Anthropic. "System Card: Claude Sonnet 5.5," section 8.3. September 28, 2026. www-cdn.anthropic.com/...205.5%20System%20Card.pdf
19. ^1 ^2 ^3Anthropic. "System Card: Claude Fable 5.1 & Claude Mythos 5.1," section 8.3. September 1, 2026. www-cdn.anthropic.com/...205.1%20System%20Card.pdf
20. ^1 ^2 ^3 ^4Google. "Introducing Gemini 3.8 Flash and 3.8 Flash Cyber." September 2, 2026. blog.google/...3-8-flash-and-3-8-flash-cyber
21. ^1 ^2 ^3 ^4Google DeepMind. "Gemini 3.8 Flash Model evaluation: Approach, methodology & results." September 2026. deepmind.google/...gemini-3-8-flash
22. ^1 ^2 ^3Artificial Analysis. "Coding Agent Index v1.5 Methodology." Accessed September 30, 2026. artificialanalysis.ai/...coding-agents-benchmarking
23. ^Mercor. "DeepSWE v1.1 leaderboard." APEX. Accessed September 30, 2026. mercor.com/...oss-deep-swe-leaderboard
24. ^1 ^2 ^3 ^4Michael Luo, Naman Jain, Jaskirat Singh, Sijun Tan, Ameen Patel, Qingyang Wu, Alpay Ariyak, Colin Cai et al. "DeepSWE: Training a Fully Open-sourced, State-of-the-Art Coding Agent by Scaling RL." Together AI, July 2, 2025. together.ai/...deepswe
25. ^Agentica. "agentica-org/DeepSWE-Preview." Hugging Face. Accessed September 30, 2026. huggingface.co/...DeepSWE-Preview