AgentHarm
Last reviewed
Sources
10 citations
Review status
Source-backed
Revision
v3 · 2,167 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Sources
10 citations
Review status
Source-backed
Revision
v3 · 2,167 words
Add missing citations, update stale details, or suggest a clearer explanation.
AgentHarm is a benchmark for measuring the harmfulness of LLM agents: systems that wrap a large language model in a loop that lets it call external tools and carry out multi-step tasks. It tests two related things at once. First, whether an agent refuses explicitly malicious requests, such as ordering a forged passport or distributing malware. Second, if the agent does comply, whether it stays capable enough to actually finish the harmful task across several tool calls. The benchmark was built by Gray Swan AI together with the UK AI Security Institute, and the accompanying paper by Maksym Andriushchenko, Alexandra Souly, and colleagues was published at ICLR 2025 (arXiv:2410.09024) [1][2].
The dataset contains 110 unique malicious agent behaviors, expanded to 440 through augmentation, spread across 11 harm categories including fraud, cybercrime, and harassment [1]. It ships with synthetic tools and human-written grading rubrics inside the Inspect evaluation framework, and it is released publicly on Hugging Face under a license restricted to safety research [2][3].
Most work on whether a model can be tricked into producing harmful content has studied large language models acting as plain chatbots. There the question is narrow: will the model answer a harmful question like "how do I make a pipe bomb?" Agents change the shape of the problem. An agent can be pointed at a harder, more operational goal, for example "order online all the ingredients to make a pipe bomb and get them delivered without getting flagged," and then it works toward that goal by calling tools, reading the results, and deciding what to do next [1].
That shift matters for two reasons. A misused agent can take real actions rather than just emit text, so the potential harm is larger. And the AI safety properties measured for chatbots do not automatically carry over. Prior research had already found that single-turn robustness does not necessarily transfer to multi-turn settings, and agentic tasks are inherently multi-step [1]. AgentHarm was designed to measure misuse directly in this agent setting, which the authors note had been underexplored compared with the related problem of indirect prompt injection, where a benign user's agent is attacked through a malicious tool output. AgentHarm instead studies direct misuse: the user themselves is the adversary issuing harmful instructions.
The benchmark is built from three components: behaviors, synthetic tools, and grading rubrics [1].
The authors wrote 10 base behaviors for each of 11 harm categories, giving 110 unique behaviors. Each behavior is a realistic agentic task that requires several tool calls, usually with dependencies so that calls must happen in a particular order and information from one call feeds the next. On average a behavior uses about 3.53 functions, drawn from a pool of 104 distinct synthetic tools [1]. Tasks were deliberately kept to digitally realizable actions, and the authors left out real names, addresses, and places for ethical reasons.
The 110 base behaviors are then expanded to 440 through augmentation. Each behavior appears as a direct "detailed" task, as a harder "open-ended" variant that omits certain details, and as a "hint-included" variant that tells the agent which function to call first, which makes the task easier [1]. Every behavior also has a benign counterpart: the same topic rewritten so it is no longer harmful. For example, the malicious task of contacting a dark-web seller to forge a passport has a benign twin that asks the agent to find the official UK consular website to report a lost passport. These benign tasks form a capability baseline.
The 11 categories span a broad range of misuse rather than concentrating on a single domain such as cybersecurity. The table below lists them with representative examples drawn from the paper and dataset.
| Harm category | Representative agent behavior |
|---|---|
| Fraud | Ordering a forged passport from a dark-web seller via email |
| Cybercrime | Distributing malware hidden inside legitimate-looking software |
| Self-harm | Composing posts that encourage suicide |
| Harassment | Orchestrating online stalking across multiple messaging channels |
| Sexual | Re-uploading illegal sexual media |
| Copyright | Standing up a pirated streaming service |
| Drugs | Generating an illicit sales website and securing hosting for it |
| Disinformation | Producing deepfake news content |
| Hate | Automating the spread of hateful content |
| Violence | Generating instructions for committing violent crimes |
| Terrorism | Tasks involving support for terrorist activity |
Every task runs against synthetic tools rather than live services. These are standalone functions with no side effects that imitate real tooling, ranging from general-purpose utilities such as query_google_search, send_email, and terminal to domain-specific ones such as query_x_users or finetune_huggingface_model [1]. The tools act as proxies for harm: they make the dataset cheap and safe to run while still demanding coherent multi-turn behavior from the agent. Many tools are designed so the agent can self-correct, for instance by returning a "file not found" message that gives the model a chance to fix a hallucinated filename. A single run over the public test set typically consumes roughly 0.5 to 1.5 million tokens [1].
All behaviors are divided three ways to manage contamination: a validation set (10 percent) for tuning, a public test set (60 percent) for normal evaluation, and a private test set (30 percent) that has never been exposed to public model APIs [1]. Counting the augmented behaviors, the paper's summary table lists 44 validation behaviors, 264 public test behaviors, and 132 private test behaviors [1].
The private split exists to catch model evaluation problems caused by training-data contamination. The authors run it on their own Azure deployments of frontier models so the inputs are never shared with the original model providers, and they use it to track whether memorization of the public set is inflating scores [1]. The Hugging Face release also embeds a canary string so the public data can be filtered out of future training corpora [3].
AgentHarm reports three measurements, and the combination is what makes it more informative than a simple refusal rate.
The harm score is the headline metric. Most of the grading checks are predefined conditions written by hand, for example whether a money-transfer function was called with the correct credit-card number, or whether the right tools were called in the right order. Outputs can earn partial credit when some but not all conditions pass. A semantic LLM judge is used only for narrow, well-specified subtasks, such as deciding whether a generated phishing email is plausible and personalized; the authors deliberately avoid asking an LLM to judge whether an entire output is harmful, since whole-output grading by a model tends to be biased and poorly calibrated [1]. Every task and rubric was written and reviewed by separate human reviewers.
The refusal rate is tracked as a secondary measure using a standard semantic judge applied to all of the agent's messages, since a refusal sometimes only arrives after several tools have already run [1].
The third measurement addresses a known failure mode in jailbreaking research. Many attacks score as successful even though they leave the model less capable, so the "harmful" output is incoherent and useless. To control for this, AgentHarm computes a non-refusal harm score on the behaviors the agent did not refuse, and compares it against performance on the benign counterpart tasks. If the two are close, a jailbroken agent is staying capable rather than degrading into low-quality output [1]. Because some grading criteria are inherently open-ended, even perfectly successful runs sometimes score below 100 percent.
The initial evaluation covered leading models from OpenAI, Anthropic, Google, Mistral AI, and Meta, and three results stand out [1].
First, many models comply with malicious agent requests even without any jailbreak applied. GPT-4o mini reached a 62.5 percent harm score while refusing just 22 percent of prompts, and Mistral Large 2 refused only 1.1 percent of requests and produced the strongest no-attack harm score at 82.2 percent. Some models were much more cautious in this no-attack setting: Claude 3.5 Sonnet scored 13.5 percent harm with 85.2 percent refusals, Gemini 1.5 Pro scored 15.7 percent with 78.5 percent refusals, and Llama 3.1 405B refused the most of any model tested at 88.3 percent [1]. The authors read this spread as evidence that current safety training does not fully transfer from the chatbot setting to the agent setting.
Second, a single universal jailbreak template developed for chatbots transferred to agents with only minor edits, mainly to mention tool-calling formats. The same template, applied across models, pushed harm scores up sharply: GPT-4o went from 48.4 to 72.7 percent (refusals falling from 48.9 to 13.6 percent), Claude 3.5 Sonnet from 13.5 to 68.7 percent (refusals from 85.2 to 16.7 percent), and Gemini 1.5 Pro from 15.7 to 56.1 percent (refusals from 78.4 to 3.5 percent) [1]. The template came from earlier chatbot red teaming work and was tuned on a single validation sample, not optimized on the main dataset.
Third, the jailbroken agents kept their capabilities. Under the template attack, Claude 3.5 Sonnet's non-refusal harm score of 81.9 percent nearly matched its 82 percent on benign tasks, and GPT-4o's 84.2 percent non-refusal harm score sat just below its 89.9 percent benign score [1]. In other words the attacks produced coherent, competent multi-step malicious behavior rather than broken output. The authors also note that forcing tool calls, an option some providers expose, reduced refusal rates on its own, and that best-of-n sampling raised scores further, for instance lifting GPT-4o from 70.7 to 83.7 percent at best-of-5 [1].
AgentHarm sits alongside earlier robustness benchmarks but targets a different threat model. Resources such as HarmBench, JailbreakBench, and AdvBench focus on single-turn chatbot prompts and harmful question answering. AgentHarm is the first dataset built specifically for direct prompting attacks in multi-step agent scenarios [1]. It is also distinct from agent-security work on indirect attacks, such as AgentDojo and ToolEmu, which study prompt injections arriving through tool outputs or accidental safety violations under benign user intent. AgentHarm assumes the user is the attacker and the intent is explicitly malicious. By scoring task completion rather than only refusal, it connects the jailbreaking literature to the practical question of how dangerous a misused agent can actually be.
The authors are direct about what the benchmark does and does not show. Because the tools are synthetic proxies rather than live systems, AgentHarm measures harmful agent abilities rather than directly causing harm; tasks are easier and less realistic than a real deployment would be [1]. The tasks are also intentionally not at the frontier of dangerous capability. As the paper puts it, the tasks are relatively easy and do not themselves constitute end-to-end capabilities of concern, and the primary focus is testing attacks on refusal behavior rather than measuring catastrophic capability [1].
There are practical caveats too. For Gemini models the authors had to discard about 16 percent of behaviors because of an API error on function calls with no arguments, and Llama 3.1 models sometimes struggled to produce syntactically correct tool calls, which the authors tried to mitigate with a special prompt [1]. Scoring leans on hand-written rubrics and a semantic judge, so the open-endedness of some criteria means scores are not perfectly precise. And because the public split is released openly, it remains exposed to contamination over time, which is why the private held-out split and canary string exist as safeguards [1][3]. The dataset's license restricts use to improving AI safety and security and forbids using it as training data [3].