Coding agent

A coding agent is a software system that uses a large language model to carry out programming work by acting on a development environment rather than only producing text for a person to copy. A coding agent reads and edits files, runs shell commands, executes tests, reads the resulting output, and revises its own work until it judges the task done or exhausts its budget. The defining property is the loop: what the model writes is executed in a real environment, and the environment's response becomes the model's next input.

That loop is what separates coding agents from the earlier generation of AI code generation tools. GitHub launched Copilot as a technical preview on 29 June 2021, describing it as an AI pair programmer that suggests lines and whole functions inside the editor; it was powered by OpenAI Codex, a GPT-3 descendant trained on natural language and public source code that OpenAI released through its API in private beta on 10 August 2021 ^[1][2]. Those systems predicted the next few lines. They did not run anything, so they had no way to find out whether their suggestion worked.

Interest in the agentic form grew out of a benchmark. In October 2023 a group of researchers released SWE-bench, a set of 2,294 software engineering problems drawn from real GitHub issues and their resolving pull requests across 12 Python repositories. Given the codebase and the issue text, a system must produce a patch that makes previously failing tests pass. The paper's best result was Claude 2 at 1.96% ^[3]. Within two years, systems built around the same task were reporting resolution rates near 80%, and coding agents had become one of the most visible commercial applications of AI agents.

What a coding agent does

The clearest published description of the loop comes from Microsoft's account of Copilot agent mode, released to Visual Studio Code Insiders on 24 February 2025. In that design the model determines the relevant context and files on its own, proposes both code edits and terminal commands, then monitors the correctness of its edits and the output of those commands and iterates to remediate problems ^[4]. OpenAI described its cloud agent in similar terms in May 2025: each task runs in an isolated sandbox preloaded with the repository, and the agent can read and edit files as well as run commands including test harnesses, linters and type checkers, iterating until tests pass ^[5].

Beneath the product differences, the components of tool use recur across implementations:

Scroll sideways for more →

Test execution is what makes the loop useful. Without an executable signal an agent is guessing; with one it can detect its own failures. Cognition's evaluation of Devin found that 72% of its passing SWE-bench runs took over ten minutes, which the company read as evidence that iteration, not first-shot accuracy, was doing the work ^[6].

Instruction files converged on a shared format faster than most of the stack. AGENTS.md, promoted as a README for agents, reports use by more than 60,000 open-source projects and support across Codex, Jules, Aider, Devin, Cursor, Gemini CLI, GitHub Copilot's coding agent, Windsurf, Zed and others ^[7].

Harnesses and scaffolds

A reported benchmark score belongs to a model plus the harness wrapped around it, not to the model alone. The 2024 SWE-agent paper made the point directly: its authors argued that language model agents are a new category of end user with their own interface needs, and that a purpose-built agent-computer interface for creating files, navigating repositories and running programs raised the pass@1 rate on SWE-bench to 12.5%, far above what non-interactive models achieved ^[8].

The pendulum has since swung back toward simplicity. The same Princeton and Stanford group released mini-SWE-agent, an agent class of roughly 100 lines of Python with no tools other than bash, no tool-calling API dependency, and a completely linear message history. Because every action runs through a plain subprocess call, it drops into any sandbox and works with any model. It reports scores above 74% on SWE-bench Verified ^[9]. Anthropic reports its own SWE-bench results using a scaffold that equips the model with only two tools, a bash tool and a string-replacement file editor, having dropped the planning tool used in earlier releases ^[10].

The SWE-bench site adopted that idea for its main board. Its SWE-bench Verified tab now defaults to a fixed-harness view, described on the site as using mini-SWE-agent "to evaluate all models with the same harness"; the underlying data file calls this the bash-only leaderboard, and it is scored on the same 500 SWE-bench Verified instances. Holding the scaffold fixed compares the underlying models rather than the engineering wrapped around them ^[9][23].

Notable systems

Scroll sideways for more →

OpenHands began life as OpenDevin in March 2024 and was renamed ^[46]; its paper describes a platform for agents that interact with the world as a human developer does, by writing code, using a command line and browsing the web, released under the MIT license ^[11]. Devin was announced by Cognition on 12 March 2024 as "the first AI software engineer" ^[12]; Devin 2.0 followed on 3 April 2025 with parallel sessions, interactive planning, codebase search and auto-generated repository wikis, on a plan starting at $20 ^[13]. Cognition signed an agreement to acquire Windsurf, the agentic IDE, on 14 July 2025, citing $82M of annual recurring revenue and more than 350 enterprise customers ^[14].

Claude Code arrived as a limited research preview alongside Claude 3.7 Sonnet on 24 February 2025, described by Anthropic as its first agentic coding tool: an active collaborator that can search and read code, edit files, write and run tests, commit and push, and use command line tools ^[15]. It became generally available with Claude 4 on 22 May 2025, adding background execution through GitHub Actions, VS Code and JetBrains extensions, and an SDK ^[10]. Google put Jules into public beta on 20 May 2025, running on Gemini 2.5 Pro inside a Google Cloud virtual machine, and took it out of beta on 6 August 2025 after developers had produced more than 140,000 publicly shared code improvements during the beta ^[16][17]. GitHub's coding agent, announced 19 May 2025, takes a different shape again: assign it an issue and it boots a virtual machine on GitHub Actions, clones the repository, analyzes the codebase with retrieval over GitHub code search, and pushes commits to a draft pull request ^[18].

Open-weight models entered the same space. Mistral released Devstral on 21 May 2025 under Apache 2.0, built with All Hands AI, reporting 46.8% on SWE-bench Verified and light enough to run on a single RTX 4090 or a Mac with 32GB of RAM ^[19].

Benchmarks

SWE-bench has grown into a family of datasets, and the scoring convention throughout is percentage of instances resolved.

Scroll sideways for more →

SWE-bench Verified is now the default citation. OpenAI worked with 93 software developers experienced in Python to screen 1,699 randomly sampled instances from the original test set, three annotators per sample, checking whether the issue was underspecified and whether the passing tests would reject valid solutions. Annotators flagged 38.3% of samples for underspecified problem statements and 61.1% for tests that might unfairly reject a correct patch; 68.3% of samples were filtered out in total, leaving 500. GPT-4o resolved 33.2% of the resulting set ^[20].

Progress on that set has been steep. The best entries dated to the first quarter of 2025 topped out at 65.4%; by December 2025 the leaders were at 79.2%. The public leaderboard, whose most recent entries are dated 26 February 2026, records the following:

Scroll sideways for more →

The leaderboard distinguishes runs "performed by or directly verified by the SWE-bench team" from self-reported submissions; of the rows above, the 74.4% mini-SWE-agent entry carries that mark and the two 79.2% entries do not ^[23]. Cost is also tracked, and it is not trivial: the 76.8% mini-SWE-agent run reports about $0.75 per instance, roughly $377 for the 500-instance set ^[23].

Vendors report their own figures under their own scaffolds. Anthropic reported 70.3% for Claude 3.7 Sonnet on SWE-bench Verified with a custom scaffold on a 489-instance subset compatible with its infrastructure, and 63.7% without that scaffold ^[15], then 72.5% for Claude Opus 4 and 72.7% for Claude Sonnet 4 ^[10]. OpenAI published 73.7% for GPT-5.1-Codex at high reasoning effort and 77.9% for GPT-5.1-Codex-Max at extra-high, alongside 58.1% on Terminal-Bench 2.0 ^[24]. For Claude Opus 4.6, released 5 February 2026, Anthropic averaged its SWE-bench Verified result over 25 trials and noted that a prompt modification produced 81.42% ^[25].

Other benchmarks fill gaps SWE-bench leaves. Aider's polyglot leaderboard runs 225 Exercism exercises across C++, Go, Java, JavaScript, Python and Rust and measures whether a model can apply edits in a well-formed way as well as solve the problem; the highest entry listed is GPT-5 at high reasoning effort with 88.0% correct, recorded in August 2025 ^[26]. Terminal-Bench, a Stanford and Laude Institute collaboration, tests terminal competence directly: version 1.0 has 80 tasks and version 2.0 has 89 across software engineering, machine learning, security and data science ^[27]. Both measure something the older function-level benchmarks such as HumanEval never touched, since those ask a model to write one self-contained function rather than operate a machine.

Limitations and criticism

Benchmark validity is the most substantive complaint. An empirical audit published in October 2024 manually reviewed the instances SWE-agent with GPT-4 had solved and found that 32.67% of successful patches involved solutions supplied directly in the issue report or its comments, and that 31.08% of passing patches were suspicious because the tests were too weak to verify correctness. After filtering both categories, the same system's resolution rate fell from 12.47% to 3.97%. The authors also observed that over 94% of the issues predated the models' knowledge cutoffs, raising data contamination concerns, and reported that the same defects appear in SWE-bench Lite and SWE-bench Verified ^[28].

Headline numbers also travel without their footnotes. Devin's widely quoted 13.86% was measured on a randomly chosen 25% of the test set, 570 of the 2,294 instances, of which Devin resolved 79, with each run capped at 45 minutes; the 1.96% baseline it was compared against came from a different setting in which a retrieval system, not the model, chose which files to edit ^[6].

Whether benchmark gains translate into faster software delivery is a separate question. METR ran a randomized controlled trial with 16 experienced open-source developers on 246 real issues in repositories they maintained, published 10 July 2025. Developers allowed to use AI tools, mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet, took 19% longer than when they were not; they had forecast a 24% speedup beforehand and still believed they had been sped up by 20% afterwards ^[29]. METR later put the confidence interval on that result at +2% to +39% ^[30]. Its February 2026 follow-up reported that the second round of the experiment had broken down: an increasing share of developers declined to participate because they did not want to work without AI, and 30% to 50% of surveyed participants said they withheld tasks for the same reason. For the ten developers who returned from the original study METR estimated a time change of -18%, meaning a speedup, with a confidence interval from -38% to +9%; for newly recruited developers the estimate was -4%. METR treats both as only very weak evidence and is redesigning the study ^[30].

Practical objections recur in day-to-day use. Review effort shifts rather than disappearing, since a human still has to read a patch nobody wrote by hand. Agents can satisfy tests without fixing the underlying defect, a form of reward hacking. Long sessions consume large numbers of tokens, and the leaderboard tracks the resulting bill explicitly, as the per-instance costs above show ^[23]. Reviewing less has acquired its own name: Andrej Karpathy coined vibe coding in February 2025 for prompting a model and accepting the result without thorough review ^[36], and Collins named it word of the year for 2025 ^[37].

Security

Giving a language model shell access and repository credentials creates an attack surface that completion tools did not have. Prompt injection is the central concern: an agent that reads issue text, web pages, dependency files or tool output can be handed instructions by whoever wrote that content.

Vendors have converged on similar controls. Claude Code documents a permission-based architecture with read-only defaults, approval prompts before Bash commands that can modify the system, a working directory boundary, a sandboxed bash tool with filesystem and network isolation, isolated context windows for web fetches so retrieved content cannot inject instructions directly, and network commands such as curl and wget excluded from auto-approval ^[31]. GitHub's coding agent can only push to branches it created, has internet access limited to a customizable allowlist, and cannot run Actions workflows without human approval; the developer who asks the agent to open a pull request cannot be the one who approves it ^[18].

The risk is not hypothetical. On 26 August 2025 attackers exploited a GitHub Actions injection flaw to steal the Nx project's npm publishing token and ship malicious package versions for about four hours. In its postmortem of 5 September 2025, Nx wrote that the malicious post-install script "scanned user systems for sensitive data, attempted to use local AI tools (like Claude and Gemini), and uploaded the results to a public GitHub repo via the GitHub CLI" ^[32]. An installed coding agent, already authenticated and already permitted to read the filesystem, is a useful instrument for an attacker who reaches the machine.

Developments in 2025 and 2026

Three shifts stand out. First, packaging: the same underlying loop now ships as a terminal program, an IDE extension, a cloud service that opens pull requests, and a code reviewer, and the leading vendors offer all of these at once. By September 2025 OpenAI had unified Codex into one product spanning the terminal, the IDE, the web, GitHub and the ChatGPT iOS app, shipping it alongside GPT-5-Codex, a version of GPT-5 optimized for agentic coding; GPT-5.2-Codex followed on 18 December 2025 and GPT-5.3-Codex on 5 February 2026, the latter reporting 77.3% on Terminal-Bench 2.0 and 56.8% on the public SWE-bench Pro set ^{[33][34][38][39]}.

Second, evaluation has moved on. With frontier systems clustered near 80% on SWE-bench Verified ^[23], attention shifted to harder and less contaminated sets: SWE-bench Pro, Terminal-Bench 2.0, and multilingual and multimodal variants. Anthropic reported that Claude Opus 4.5 led on 7 of the 8 programming languages it measured on SWE-bench Multilingual ^[35], and Multi-SWE-bench, released in April 2025, extends the same idea to Java, TypeScript, JavaScript, Go, Rust, C and C++ ^[45].

Third, models that publish their weights closed part of the gap. On the bash-only board, which fixes the scaffold, the highest entry the leaderboard flags as an open-weights model is Kimi K2 Thinking at 63.4%, ahead of MiniMax M2 at 61.0%, DeepSeek V3.2 Reasoner at 60.0% and Devstral small at 56.4%, against 76.8% for the top entry overall ^[23]. Devstral is Apache 2.0 licensed and small enough for a single consumer GPU ^[19], which matters for teams that cannot send proprietary code to a hosted API.

Two questions remain unsettled. Vendors report large internal gains: OpenAI says 95% of its engineers use Codex weekly and that those engineers ship roughly 70% more pull requests since adopting it ^[24]. The best independent field measurement, meanwhile, still cannot cleanly separate a real speedup from the selection effects that adoption itself creates ^[30]. And the review bottleneck has not moved: however many pull requests an agent opens, each one still needs a person willing to be accountable for merging it.

See also

• SWE-bench
• SWE-agent
• Claude Code
• OpenHands
• Devin
• Harness

References

1. GitHub Blog, "Introducing GitHub Copilot: your AI pair programmer", Nat Friedman, 29 June 2021. https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/ ↩
2. OpenAI, "OpenAI Codex", 10 August 2021 (updated 16 May 2025). https://openai.com/index/openai-codex/ ↩
3. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", arXiv:2310.06770, 10 October 2023. https://arxiv.org/abs/2310.06770 ↩
4. Visual Studio Code Blog, "Introducing GitHub Copilot agent mode (preview)", Isidor Nikolic, 24 February 2025. https://code.visualstudio.com/blogs/2025/02/24/introducing-copilot-agent-mode ↩
5. OpenAI, "Introducing Codex", 16 May 2025. https://openai.com/index/introducing-codex/ ↩
6. Cognition, "SWE-bench technical report", 15 March 2024. https://cognition.com/blog/swe-bench-technical-report ↩
7. AGENTS.md project site. https://agents.md/ ↩
8. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press, "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering", arXiv:2405.15793, 6 May 2024. https://arxiv.org/abs/2405.15793 ↩
9. mini-SWE-agent repository README, SWE-agent organization. https://github.com/SWE-agent/mini-swe-agent ↩
10. Anthropic, "Introducing Claude 4", 22 May 2025. https://www.anthropic.com/news/claude-4 ↩
11. Xingyao Wang et al., "OpenHands: An Open Platform for AI Software Developers as Generalist Agents", arXiv:2407.16741, 23 July 2024. https://arxiv.org/abs/2407.16741 ↩
12. Cognition, "Introducing Devin, the first AI software engineer", 12 March 2024. https://cognition.com/blog/introducing-devin ↩
13. Cognition, "Devin 2.0", 3 April 2025. https://cognition.com/blog/devin-2 ↩
14. Cognition, "Cognition's acquisition of Windsurf", Scott Wu, 14 July 2025. https://cognition.com/blog/windsurf ↩
15. Anthropic, "Claude 3.7 Sonnet and Claude Code", 24 February 2025. https://www.anthropic.com/news/claude-3-7-sonnet ↩
16. Google, "Build with Jules, your asynchronous coding agent", Kathy Korevec, 20 May 2025. https://blog.google/technology/google-labs/jules/ ↩
17. Google, "Jules, our asynchronous coding agent, is now available for everyone", 6 August 2025. https://blog.google/technology/google-labs/jules-now-available/ ↩
18. GitHub Blog, "GitHub Copilot: Meet the new coding agent", Thomas Dohmke, 19 May 2025. https://github.blog/news-insights/product-news/github-copilot-meet-the-new-coding-agent/ ↩
19. Mistral AI, "Devstral", 21 May 2025. https://mistral.ai/news/devstral ↩
20. OpenAI, "Introducing SWE-bench Verified", 13 August 2024. https://openai.com/index/introducing-swe-bench-verified/ ↩
21. John Yang et al., "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?", arXiv:2410.03859, 4 October 2024. https://arxiv.org/abs/2410.03859 ↩
22. Xiang Deng et al., "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?", arXiv:2509.16941, 21 September 2025. https://arxiv.org/abs/2509.16941 ↩
23. SWE-bench leaderboards, swebench.com; underlying data file data/leaderboards.json in the site repository. https://www.swebench.com/ and https://github.com/SWE-bench/swe-bench.github.io/blob/master/data/leaderboards.json ↩
24. OpenAI, "Building more with GPT-5.1-Codex-Max", 18 November 2025. https://openai.com/index/gpt-5-1-codex-max/ ↩
25. Anthropic, "Introducing Claude Opus 4.6", 5 February 2026. https://www.anthropic.com/news/claude-opus-4-6 ↩
26. Aider LLM Leaderboards, polyglot coding benchmark. https://aider.chat/docs/leaderboards/ ↩
27. Terminal-Bench, Stanford and Laude Institute. https://www.tbench.ai/ ↩
28. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, Song Wang, "SWE-Bench+: Enhanced Coding Benchmark for LLMs", arXiv:2410.06992, 9 October 2024. https://arxiv.org/abs/2410.06992 ↩
29. METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", Joel Becker, Nate Rush, Beth Barnes, David Rein, 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ ↩
30. METR, "We are Changing our Developer Productivity Experiment Design", 24 February 2026. https://metr.org/blog/2026-02-24-uplift-update/ ↩
31. Anthropic, Claude Code documentation, "Security". https://code.claude.com/docs/en/security ↩
32. Nx Blog, "S1ngularity: What Happened, How We Responded, What We Learned", Juri Strumpflohner, 5 September 2025. https://nx.dev/blog/s1ngularity-postmortem ↩
33. OpenAI, "Introducing upgrades to Codex", 15 September 2025. https://openai.com/index/introducing-upgrades-to-codex/ ↩
34. OpenAI, "Introducing GPT-5.2-Codex", 18 December 2025. https://openai.com/index/gpt-5-2-codex/ ↩
35. Anthropic, "Introducing Claude Opus 4.5", 24 November 2025. https://www.anthropic.com/news/claude-opus-4-5 ↩
36. Wikipedia, "Vibe coding". https://en.wikipedia.org/wiki/Vibe_coding ↩
37. Collins Dictionary, "The Collins Word of the Year 2025". https://www.collinsdictionary.com/woty ↩
38. OpenAI, "Introducing GPT-5.3-Codex", 5 February 2026. https://openai.com/index/introducing-gpt-5-3-codex/ ↩
39. OpenAI, Codex changelog (release dates for GPT-5.1-Codex-Max, GPT-5.2-Codex and GPT-5.3-Codex). https://learn.chatgpt.com/docs/changelog ↩
40. SWE-agent repository, Princeton NLP and collaborators (MIT license). https://github.com/SWE-agent/SWE-agent ↩
41. Aider repository, Aider-AI (Apache 2.0). https://github.com/Aider-AI/aider ↩
42. Google for Developers Blog, "The next chapter of the Gemini era for developers", 11 December 2024. https://developers.googleblog.com/en/the-next-chapter-of-the-gemini-era-for-developers/ ↩
43. Cline repository, Cline (Apache 2.0). https://github.com/cline/cline ↩
44. Gemini CLI repository, Google (Apache 2.0). https://github.com/google-gemini/gemini-cli ↩
45. Daoguang Zan et al., "Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving", arXiv:2504.02605, 3 April 2025. https://arxiv.org/abs/2504.02605 ↩
46. OpenHands repository (created March 2024 as OpenDevin). https://github.com/OpenHands/OpenHands ↩