Real-SWE

Real-SWE is a coding-agent benchmark published in September 2026 by Specific Labs, a San Francisco company that licenses operational data and source code from businesses and packages them as training and evaluation material for AI labs. The published leaderboard scores eight model-and-harness pairs on ten tasks drawn from private production codebases that Specific licensed from real companies; the report presents those ten as a sample of a larger task set and does not give the benchmark's full size, on the argument that benchmarks mined from public repositories can no longer separate problem solving from recall. Eight model-and-harness pairs were scored across 640 rollouts. The strongest, Claude Fable 5.1 running in Claude Code, resolved 38.8% of tasks; the weakest, GPT-5.6 Sol in Codex CLI, resolved 16.2%.^[1]

The report is bylined Snagnik Das, Siddhant Paliwal and Janak Sunil, and is published at withspecific.com/benchmarks/real-swe and mirrored at realswe.withspecific.com.^[1][3] Specific announced it as a Y Combinator launch on September 10, 2026, and it was submitted to Hacker News on September 12, where it had collected 274 points and 156 comments as of September 15, 2026.^[2][3]

Why private codebases

Specific's stated motivation is contamination and distribution. Public agentic coding benchmarks in the SWE-bench lineage are built by mining merged pull requests from open-source repositories, so the fixes and the discussion around them plausibly sit in pretraining data. The company frames Real-SWE tasks as "natively out of distribution" because the code and its solutions are not published anywhere, and claims that 99% of the tokens inside real enterprises are invisible to frontier models.^[1]

That complaint is not unique to Specific. Two 2026 benchmarks it measures itself against take different routes around the same problem: DeepSWE writes 113 original tasks across open-source repositories and never contributes them upstream, and reports that an independent LLM judge disagrees with its hand-written verifiers about a tenth as often as with the tests SWE-Bench Pro inherits from merged patches (1.4% against 32.4%).^[4] Cognition's FrontierCode instead keeps the repositories public but has maintainers write tasks and rubrics for their own projects, and argues that SWE-bench Verified and SWE-Bench Pro test functional correctness rather than whether a maintainer would merge the change.^[5] Real-SWE takes the third route: keep the grading conventional, and make the repository itself unavailable.

Two further arguments appear in the report. The tasks are economically real: each one was assigned to a salaried engineer and has a direct relationship to company spend, rather than testing an experimental capability. And they are company-specific: Specific argues that models are weaker at absorbing an organization's own conventions, and that its results show current agents "frequently miss requirements or don't verify their assumptions."^[1]

In the launch post, the founders put the same point in customer terms: the number companies care about is whether an agent "can land a fix in our billing system, our permissions layer, our customer data pipeline, without having seen any of it before."^[3]

Tasks and environments

Specific says it screened candidate codebases for real usage, engineering quality and production load. The report identifies three of the contributing products only by description: a Luma or Partiful competitor with more than 200,000 users and a top-100 App Store ranking, a consumer fintech platform processing more than 100,000 bank statements, and enterprise AI sales platforms.^[1]

Each task runs in an isolated sandbox that exposes only the services its workflow needs. Across the task set those include an AWS emulator, Docker, Kubernetes, GitHub, a Linear MCP server, PostgreSQL, MySQL, MongoDB, Gel, Redis, Go, Python, Node.js, Vitest, Slack, Intercom, Google Drive, email and ClickUp.^[1]

One task instruction is published in full, as a worked example. It asks the agent to fix invoice billing so that each business on a multi-tenant platform charges the right tax and exempt customers are not taxed, with each tenant configured differently: some hold their own rate, some price every invoice against the buyer's destination through a tax-authority provider, and some collect nothing. The agent has to call the authority with both addresses, the priced lines and the tenant's product category, choose the sandbox or production authority according to the tenant's account, report an address the authority rejects without aborting the invoice, write the rate, tax and gross onto the issued invoice, file the settled sale back under the invoice number, and show both parties' VAT registrations on European invoices. The sandbox for that task exposes a TaxJar sandbox, a TaxJar production endpoint, an InfluxDB ledger and a NestJS service in TypeScript.^[1]

Methodology

All tasks are in Harbor format. Agents run in isolated sandboxes, and verifiers are injected only at grading time, so the agent never sees the tests it will be graded against. Specific says the verifiers are inspired by the codebase's existing test suites or reuse those tests verbatim.^[1]

Rather than run every model through one common scaffold, Specific used what it calls native harnesses, on the grounds that this is how enterprise engineers actually use them. In practice that meant each provider's own agent tool for six of the eight entries and Claude Code for GLM-5.3, the harness Z.ai itself evaluates that model in. The consequence, which the report states plainly, is that Real-SWE scores model-and-harness combinations and not models in isolation. All models were run at high reasoning effort.^[1] Sunil repeated both points on Hacker News in reply to a commenter who had missed the harness column, adding that the release is "still v1."^[2]

Each model ran eight independent rollouts on each of the ten tasks, for 80 rollouts per model and 640 in total. The headline resolution rate is pass@1 averaged over those eight runs, and the leaderboard shows 95% confidence intervals.^[1]

The report also measures how specified its instructions are, against three other benchmarks:

The FrontierCode and DeepSWE figures are taken from Cognition's published FrontierCode comparison, which gives medians of 2,056 characters for a FrontierCode prompt including codebase guidelines, 1,975 for DeepSWE, and 6 files touched by the reference patch in both.^[5] Specific measured the Terminal-Bench 3 and FrontierSWE v2 figures itself from those benchmarks' task files (74 and 34 tasks respectively), and measured its own from eight repository-backed sample tasks rather than all ten.^[1][6][7]

Results

Source: Real-SWE report.^[1] Resolution rate is pass@1 averaged over eight runs per task; failed-rollout counts are the denominators the report uses in its failure-mode breakdown.

Two placements cut against the usual ordering. GLM-5.3, a model from the Chinese lab Z.ai, was run in Claude Code and placed fifth, ahead of both Meta's Muse Spark 1.3 and OpenAI's GPT-5.6 Sol. And Gemini 3.8 Flash, a low-cost tier model, placed fourth at the lowest measured cost per rollout.

Task-by-task results

The report publishes every cell of the task-by-model grid. Six of the ten tasks sit below a 15% resolution rate, and one is unsolved by every model.

Source: Real-SWE report.^[1] No model solves all ten tasks, and the rankings invert task by task: Grok 4.6 is the only model to pass the linearizable scan more than twice, while Kimi K3 fails every one of its eight attempts at the easiest task in the set.

Failure modes

Specific classified each failed rollout using a five-label subset of the verdict-tag taxonomy from DeepSWE, the Datacurve benchmark of original long-horizon engineering tasks.^[1][4] The labels are: unverified assumption (building on a guess instead of checking the workspace), missed requirement (leaving out behavior the instruction requires), integration error (right idea, wired into the surrounding system wrongly), regression (breaking existing behavior), and wrong file (delivering the change somewhere the running application never calls).^[1]

Percentages below are shares of each model's own failed runs, not of all runs.

Source: Real-SWE report.^[1]

Missed requirements are the largest single bucket across the field, and outright regressions are rare. The spread between models is wide: GPT-5.6 Sol fails on unverified assumptions four times as often as Gemini 3.8 Flash does, while Gemini 3.8 Flash fails on integration three times as often as GPT-5.6 Sol. Only GLM-5.3 and Kimi K3 ever put the change in a file the application never calls.

The report also breaks failures down by wall-clock time, and finds that short rollouts are not much safer than long ones: 76 of 108 rollouts that finished in under ten minutes failed (70.4%), against 385 of 532 that took ten minutes or longer (72.4%). The prose above that chart quotes the two shares as 71.4% and 73.4%, which do not match the counts in the chart's own labels.^[1]

Cost and effort

Specific publishes an estimated dollar cost per rollout alongside mean output tokens, and notes that the relationship to score is weak.

Source: Real-SWE report.^[1] The Kimi K3 figure carries a note that usage data was incomplete and the real cost may be higher.

The most and least expensive configurations are separated by a factor of 2.8 in price, but the second-cheapest configuration finishes last and the third-cheapest finishes third. Token spend is likewise uninformative about score: GLM-5.3 emits roughly five times as many output tokens per rollout as GPT-6 Astra and finishes below it.

Reception

Hacker News commenters split between practitioners whose own experience matched the ranking and readers who objected to the design.

The thread's leading comment, from springtimesun, described building an equivalent private harness by hand: rewind a repository to the commit before a change, sandbox the agent with only what it had then, feed it the original ticket, and grade against the accepted pull request. The commenter recommended the exercise and reported that on Rails and front-end work most tested models could reach a mergeable pull request with a human in the loop.^[2]

Contamination was the most common objection, aimed not at the task sources but at what happens once they are used. ignoramous wrote that "once we send code & prompts to the providers, it can no longer be considered private," and lmeyerov, who says he runs the botsbench.com evaluation, argued that many large private codebases are already not private to Claude Code and Codex, and that a benchmark should measure contamination on every run. matt3210 made the shorter version of the point: the tasks will end up in the training set eventually. Asked directly whether the licensed codebases had been handed to OpenAI and Anthropic, Sunil replied that they had not; dgellow replied that this still amounts to private codebases uploaded to the AI labs.^[2]

traceroute66 called the result "pinky-promise benchmarking" on reproducibility grounds and doubted that a company careful enough to keep code off public forges would then license it for agent evaluation. kadoban answered that a private benchmark trades transparency for resistance to gaming, and sigmar noted that Artificial Analysis and ARC-AGI already ask readers to trust the tester. barbegal objected that without a human baseline there is no way to tell a weak model from a badly specified task, and read the sample instruction as underspecified enough to explain the assumption and requirement failures. evalmaster123 raised the same concern in blunter terms, pointing at an unanchored "Monday" in the published prompt and calling the release untrustworthy without a released sample. In a neighboring subthread about home-built benchmarks, bisonbear set out the grading questions such an effort has to answer, including whether an LLM grader is calibrated and whether it favors agents built on the same model.^[2]

skhameneh said the results were an outlier against personal experience and against Artificial Analysis, specifically on Kimi K3's token consumption and cost, and initially faulted the report for not stating harnesses or reasoning levels before correcting himself on the harness column. Sunil replied with the native-harness and high-reasoning answer and called the release a first version.^[2]

Gemini 3.8 Flash's fourth place drew the most disagreement. freakynit, hollars and obilgic each said it matched their own use, while bdlowery and cute_boi treated it as evidence the benchmark is unsound, describing the model looping over the same files across hundreds of tool calls. glub found the failure-mode chart counterintuitive in the opposite direction, having expected Fable, not Sol and Astra, to lead on unverified assumptions.^[2]

In the thread Sunil said the licensed codebases were written before 2023, so their contents predate competent AI coding assistants; that Specific manually vets every codebase and company; and that the company intends to open-source some tasks and model trajectories.^[2]

Limitations

The report's own framing and the criticism of it converge on several constraints that any reader of the leaderboard has to carry.

The published results cover ten tasks, which the report describes as a sample of the benchmark rather than the whole of it. On that leaderboard a single task is worth 10 percentage points of a model's score, so the gap between first and second place, 5 points, is smaller than one task. Eight rollouts per task keeps sampling noise visible but does not make the task set representative, and Specific publishes confidence intervals rather than claiming separation between adjacent ranks.

The measurements are of model-and-harness pairs. Claude Code appears twice, with two different models, and GLM-5.3's fifth place is a result for GLM-5.3 driven by Anthropic's agent scaffold, not for GLM-5.3 as such. Nothing in the report isolates the harness contribution.

Nobody outside Specific can rerun it. The codebases are licensed and private, the task set is not released, beyond a sample Specific releases on request through a form on the report page,, and the verifiers are not published, so the leaderboard rests on the publisher's account of its own procedure. Sunil's statement that some tasks and trajectories would be open-sourced was made in a comment thread rather than in the report itself.

Finally, Specific sells what it is measuring. The company's business is buying and licensing company data for AI training and evaluation: its site tells prospective sellers their data "could be worth $100K-$1M" and offers up to $10,000 for a referral that leads to a purchase, and the launch post invites labs that want "the full task set for training" to email the chief executive directly.^[3][8][9] A benchmark whose headline finding is that the best frontier agent resolves under 40% of real enterprise tasks is also an argument for buying the training data that would close the gap, and Real-SWE should be read with that alignment of interests in view.

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18Introducing Real-SWE - Snagnik Das, Siddhant Paliwal and Janak Sunil, Specific Labs, September 2026 (mirrored at realswe.withspecific.com).
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases - Hacker News discussion, submitted September 12, 2026.
3. ^1 ^2 ^3 ^4Real-SWE: A coding benchmark built from private company codebases - Specific Labs launch post, Y Combinator, September 10, 2026.
4. ^1 ^2Wenqi Huang, Charley Lee, Leonard Tng and Serena Ge, DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks, arXiv:2607.07946, July 8, 2026.
5. ^1 ^2Introducing FrontierCode - Cognition, June 8, 2026.
6. ^Terminal-Bench task directory - the commit Specific measured, containing 74 task directories.
7. ^FrontierSWE v2 task directory - 34 tasks.
8. ^Company data partnerships - Specific Labs.
9. ^Refer a company - Specific Labs.