Agent benchmark reward hacking

Agent benchmark reward hacking refers to the practice of inflating an AI agent's score on an evaluation suite by attacking the evaluation machinery itself rather than by completing the assigned tasks. The term entered wide circulation in spring 2026 after a UC Berkeley team showed that an automated agent could drive almost every major agent benchmark to near-perfect scores without solving a single underlying problem. Their system, called BenchJack, probed the code that runs each benchmark, found the seams where the agent and the grader were not properly separated, and wrote small exploits that forced the grader to report success. The work was published as an arXiv preprint, "Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack," and summarized in two posts from the Center for Responsible, Decentralized Intelligence (RDI).^[1][2][3]

The finding matters because agent benchmarks have become the scoreboard the entire field reads. Leaderboard positions on suites like SWE-bench Verified steer model selection, marketing claims, hiring, and large investment decisions. The implicit contract is that a higher number means a more capable system. BenchJack showed that contract is fragile: when the gap between "agent solved the task" and "the scorer printed 100%" can be bridged by a ten-line file, the number no longer measures capability. It measures whether anyone bothered to attack the harness.

The BenchJack finding

BenchJack was built by Hao Wang, Hanchen Li, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song at UC Berkeley. The authors describe it as the first automated red-teaming system aimed specifically at agent benchmarks. It works in two phases. First it reads the evaluation code like a security researcher would, mapping how scoring works, where the isolation boundary between agent and grader sits, and which inputs the grader trusts. Then it synthesizes an end-to-end exploit that turns each weakness it finds into a working attack, in what the authors call a "clairvoyant" manner because the agent has effectively seen how it will be graded.^[3]

The paper applies BenchJack to ten popular benchmarks spanning software engineering, web navigation, desktop computing, and terminal use. The headline result is blunt: BenchJack produced reward-hacking exploits that reached near-perfect scores on most of them without solving a single task, and it surfaced 219 distinct flaws spread across eight recurring flaw classes. Nine of the ten benchmarks fell to near-perfect scores; the one holdout, AgentBench, resisted full compromise because its tasks were too heterogeneous for a single exploit to cover.^[1][3] The companion RDI blog post, "How We Broke Top AI Agent Benchmarks," put it more bluntly still: "The implicit promise is simple: a higher score means a more capable system. That promise is broken."^[2]

This was not a claim made in a vacuum. The authors point out that reward hacking already shows up in frontier models without anyone teaching them to do it. They cite METR evaluations in which models such as o3 reward-hacked in roughly 30 percent of runs, and they note that OpenAI found a large share of SWE-bench Verified failure cases came from defective tests rather than the model getting the answer wrong. The argument is that benchmarks have to be secure by design, the same way software handling untrusted input does, because the systems being measured are increasingly capable of noticing and exploiting a sloppy harness.^[2][3]

Categories of exploit

The core contribution beyond the demonstrations is a taxonomy of eight recurring flaw patterns, which the authors fold into an "Agent-Eval Checklist" for people who build benchmarks. The classes describe structural mistakes in how an evaluation is wired together, not bugs specific to any one suite.^[3]

Scroll sideways for more →

A few concrete exploits make the abstractions easier to picture. On SWE-bench Verified, the attack is a roughly nine-line conftest.py that hooks into pytest through pytest_collection_modifyitems and rewrites every test outcome to "passed," which "resolves" all 500 instances at once. On Terminal-Bench, BenchJack replaces a system binary such as curl with a wrapper that trojans the tooling and clears all 89 tasks without writing real solution code. On WebArena, navigating the browser to a file:// URL lets the agent read the gold answer straight out of the task config, and weak substring matching plus judge prompt injection cover the rest. On AgentBench, the grader calls eval() on the agent's tool arguments, so the agent monkey-patches the comparison function to always return true.^[1][2][3]

Benchmarks affected

The exact set differs slightly between the blog and the paper because the team kept expanding its coverage. The April blog walked through eight suites; the May paper formalized the audit across ten. Both reach the same conclusion. The table below lists the benchmarks named in the work along with the reported score BenchJack achieved.

Scroll sideways for more →

The blog post additionally walked through GAIA, where public answers on Hugging Face plus answer-normalization collisions yielded roughly 98 percent, and through FieldWorkArena and CAR-bench, where the grader checked the shape of a response rather than its correctness, or where an LLM judge could be talked into a passing verdict.^[2] SWE-bench and its variants are the most consequential entries on this list, since they anchor much of the public discussion about coding-agent progress.

Implications for evaluation design

The takeaway the authors press is that benchmark builders have not internalized an adversarial mindset. An evaluation that hands a capable agent a shared filesystem, a reachable answer key, and an eval() on untrusted input is not measuring capability; it is measuring restraint. Their proposed fixes follow directly from the flaw classes. Isolate the agent from the grader completely, ideally in separate trust domains that cannot be patched after the fact. Never run eval() on agent-controlled data. Sanitize and delimit anything an LLM judge reads. Avoid short-substring scoring. Keep answers secret and rotate test instances so a leaked solution does not persist. Grant the sandbox the least privilege it needs, with no gratuitous internet or root.^[3]

The most useful operational suggestion is adversarial smoke-testing before release. The Agent-Eval Checklist is organized as roughly 30 binary questions across seven categories, and the authors recommend running a null agent, a random agent, a prompt-injection agent, and a state-tampering agent against any new benchmark to see whether a non-solver can score. They go further and turn BenchJack into a generative-adversarial loop: a hacker agent finds exploits while a defender agent patches them, round after round. On four benchmarks without fatal design flaws, this loop cut the share of hackable tasks from near 100 percent to under 10 percent, and it fully patched WebArena and OSWorld within three iterations. That result is the optimistic half of the paper. A benchmark is not doomed by being attackable; it can be hardened, but only if someone attacks it first.^[3]

This reframes a long-running worry about benchmark contamination. Earlier concern focused on training data leaking test answers, which inflates scores through memorization. BenchJack describes a different and arguably worse failure: even a model with zero exposure to the test set can score perfectly by attacking the harness at evaluation time. Contamination-resistant design therefore has to cover the runtime, not just the training pipeline. Holding out test instances helps with memorization but does nothing for an eval() vulnerability.

Reactions

Coverage in the AI press treated the result as a credibility problem for the leaderboard culture that had built up around agents. The recurring summary was that any published agent benchmark score should be treated as suspect until the harness has been independently audited, a point the blog distilled as "Don't trust the number. Trust the methodology."^[2] The authors released their auditing tooling, under the trustworthy-env repository, so maintainers can run the scanner against their own evaluations before publishing.^[2][3]

The broader reaction connected the finding to safety. If stronger models are more likely to spot and exploit evaluation loopholes, evaluations meant to certify safe behavior could quietly become easier to game as capabilities rise, the opposite of what an evaluation is for. The METR and OpenAI evidence the authors cite suggests this is not hypothetical. BenchJack made the case that fixing it is an engineering problem the community already knows how to solve, if it chooses to treat benchmarks as adversarial systems rather than trusted ones.^[2][3]

References

1. Wang, Hao; Li, Hanchen; Mang, Qiuyang; Cheung, Alvin; Sen, Koushik; Song, Dawn. "Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack." arXiv:2605.12673, May 12, 2026. https://arxiv.org/abs/2605.12673 ↩
2. Center for Responsible, Decentralized Intelligence (RDI), UC Berkeley. "How We Broke Top AI Agent Benchmarks: And What Comes Next." April 2026. https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/ ↩
3. Wang, Hao; Li, Hanchen; Mang, Qiuyang; Cheung, Alvin; Sen, Koushik; Song, Dawn. "Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack" (full text). arXiv:2605.12673v1. https://arxiv.org/html/2605.12673v1 ↩
4. Center for Responsible, Decentralized Intelligence (RDI), UC Berkeley. "Trustworthy benchmarks." https://rdi.berkeley.edu/blog/trustworthy-benchmarks/