# Harness (AI)

> Source: https://aiwiki.ai/wiki/harness
> Updated: 2026-07-14
> Categories: AI Agents, AI Benchmarks, Model Evaluation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution to "AI Wiki (aiwiki.ai)".

_This article is about software that wraps and evaluates AI models. For the 2015 paper "Explaining and Harnessing Adversarial Examples" and other uses of the word, see the section on other meanings below._

**Harness**, in artificial intelligence, is the software scaffolding that wraps a machine learning model, most often a [large language model](/wiki/large_language_model), so the model can be put to work for a purpose beyond answering a single prompt. The term covers two closely related but distinct kinds of system. An **agent harness**, also called an AI harness, is the runtime layer that turns a model into an [AI agent](/wiki/agent): it supplies the loop, the [tools](/wiki/tool_use), the memory, and the guardrails that let the model take actions in an environment. An **evaluation harness**, or eval harness, is the framework that administers [benchmarks](/wiki/benchmark) to a model: it formats prompts, runs the model over a task set, extracts and scores the answers, and reports the results. Both borrow their name and their central metaphor from the test harness of classical software engineering, the rig that drives a piece of code under controlled conditions and observes what it does.[9][31]

The common thread is a division of labor: the model supplies the intelligence, and the harness supplies almost everything else needed to use that intelligence or to measure it. Anthropic describes an agent harness as "the software scaffolding around a model: the loop, tools, context management, and guardrails that turn raw intelligence into a working agent."[1] Databricks frames the split as anatomy, calling the model the "brain" and the harness "everything around it that helps the agent operate safely and reliably."[2] A recurring lesson across both senses is that a reported capability number is a property of the model and its harness together, not of the model alone: the same model can look markedly stronger or weaker depending on the harness wrapped around it.[5][2][29]

## Etymology and origins

The word "harness" entered English around 1300, meaning personal fighting equipment or the trappings of a war-horse, from Old French. The sense of straps and fittings for a draught animal followed in the early fourteenth century, and the figurative verb "to harness," meaning to control a force and put it to useful work, dates from the 1690s.[34] That horse-tack image of channeling a power source is the one every later technical sense reuses.

In software engineering, a test harness is the automation code that exercises a program under test by feeding it inputs and comparing the outputs against expected values. It is usually described either as a test execution engine paired with a repository of test scripts, or as the stubs and drivers that stand in for parts of a system that are not yet available.[31][32] Artificial intelligence borrowed the term in stages. The evaluation sense came first: [EleutherAI](/wiki/eleutherai) released its lm-evaluation-harness in 2021 as "a framework for few-shot evaluation of language models," and "harness" became standard vocabulary for the rig that runs a model over benchmark tasks.[11][12] The agent sense arrived with the autonomous-agent wave of 2023 and 2024, when the wrapper that lets a model act in a loop was more often called a "scaffold." [METR](/wiki/metr) used "agent scaffolding" from early 2024, and the [SWE-agent](/wiki/swe_agent) project named its version the Agent-Computer Interface.[22][10] "Agent harness" became the mainstream label over 2025 and 2026, as [Anthropic](/wiki/anthropic) described its Claude Agent SDK as "a powerful, general-purpose agent harness" and a wider practitioner literature grew up around designing these systems.[6][9] The concept predates the word: Anthropic's influential December 2024 essay "Building Effective Agents" describes the agent loop in detail without using either "harness" or "scaffold."[4]

## Agent harnesses

### The agent loop

At the core of every agent harness is a loop. The model is given a task and a set of tools; it responds with an action, such as a tool call; the harness executes that action, captures the result, and returns the result to the model as a new observation; and the cycle repeats until the model signals that it is finished or a stop condition is reached. Anthropic's working definition of an agent is exactly this arrangement: agents "are typically just LLMs using tools based on environmental feedback in a loop."[4] The same essay draws a useful line between workflows, "systems where LLMs and tools are orchestrated through predefined code paths," and agents, "systems where LLMs dynamically direct their own processes and tool usage."[4] A harness can run either, but the agent case is what makes the harness load-bearing, because the model, not the programmer, decides the sequence of steps.

### What a harness contains

Beyond the loop, a production agent harness bundles several services around the model. Databricks enumerates eight common building blocks: system prompts; tools and tool execution; sandboxes and execution environments; a filesystem and durable storage; memory and context management; feedback loops and self-verification; guardrails and human-in-the-loop controls; and observability and logging.[2] Context management is repeatedly singled out as the hard part for long-running agents, because the model's [context window](/wiki/context_window) is finite and a long task can overflow it. Anthropic's [Claude Agent SDK](/wiki/claude_agent_sdk), for instance, "has context management capabilities such as compaction, which enables an agent to work on a task without exhausting the context window," a challenge that has made [context engineering](/wiki/context_engineering) a discipline in its own right.[6]

### Notable agent harnesses

Many widely used coding and agent tools are, structurally, harnesses wrapped around a general-purpose model.

| Harness | Developer | Notes |
| --- | --- | --- |
| [Claude Code](/wiki/claude_code) and the [Claude Agent SDK](/wiki/claude_agent_sdk) | Anthropic | Command-line coding agent; the SDK exposes the same harness for general tasks |
| [OpenAI Codex CLI](/wiki/openai_codex) | OpenAI | Terminal-based coding agent |
| [Aider](/wiki/aider) | Open source | Terminal pair-programming assistant that edits a local git repository |
| [Cline](/wiki/cline) | Open source | Coding agent that runs inside the VS Code editor |
| [OpenHands](/wiki/openhands) | All Hands AI | Open-source agent platform, formerly OpenDevin |
| [SWE-agent](/wiki/swe_agent) | Princeton NLP | Research harness that introduced the Agent-Computer Interface |
| [Devin](/wiki/devin) | Cognition | Commercial autonomous software-engineering agent |
| [Auto-GPT](/wiki/autogpt) | Open source | One of the first popular autonomous agents built on GPT-4 (2023) |

### Harness, scaffold, framework, and orchestrator

The vocabulary here is not fully settled. In loose usage, "harness" and "scaffold" are interchangeable and both mean roughly everything around the model. [Hugging Face](/wiki/hugging_face)'s agent glossary proposes a finer split: scaffolding is "the behavior-defining layer around the model," meaning the system prompt, the tool descriptions, and how responses are parsed and remembered, while the harness is "the execution layer inside the agent: it calls the model, handles its tool calls, decides when to stop."[3] In that framing, scaffolding is what the model works from and the harness is what runs it. A 2026 analysis by Sanderson de Macedo goes further, setting out necessary and sufficient conditions for a system to count as an agent harness: an agent loop that interleaves reasoning, action, and observation; a tool interface; active context management; and at least one control mechanism that operates independently of the model. Under those conditions an agent [framework](/wiki/langchain) such as LangChain, a plain SDK, an evaluation harness, and a step-by-step orchestrator each fall short in a specific way, because none of them closes an adaptive loop around the model at runtime.[9]

## Why the harness matters: model versus harness

The single most consequential fact about harnesses is that they change results. Because a benchmark like [SWE-bench](/wiki/swe_bench) runs a whole agent rather than a bare model, the reported score reflects the scaffolding as much as the network. As Anthropic's Erik Schluntz put it, "SWE-bench doesn't just evaluate the AI model in isolation, but rather an entire 'agent' system," and "the performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model."[5] Anthropic reached 49% on SWE-bench Verified with a deliberately minimal scaffold, just a prompt, a bash tool, and a file-editing tool, a design meant to hand as much control as possible to the model.[5]

The academic literature makes the point under controlled conditions. The SWE-agent paper holds the language model fixed and varies only the interface between the agent and the computer: with GPT-4 Turbo unchanged, its Agent-Computer Interface lifted the resolved share of SWE-bench Lite to 18%, several times the rate of a non-interactive retrieval baseline and far above the previous state of the art.[10] METR, which evaluates frontier models for dangerous capabilities, treats this dependence as central to its method. To avoid understating what a model can do, and therefore understating its risks, it instructs evaluators to give a model "the best available scaffolding" and tooling, and it calls the broader practice of tuning that setup to draw out a model's peak performance elicitation.[22][24] In one study, moving from the base GPT-4 to its post-trained version raised agent performance by 26 percentage points, and further scaffolding work added about 8 more, though that last increment was not statistically significant.[23]

As base models converge in raw capability, this layer matters more rather than less. Databricks observes that "as models converge in raw capability, the harness increasingly determines performance."[2] The 2026 benchmark study Harness-Bench argues, on the strength of these effects, that "agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone."[29]

There is a flip side. Because every part of a harness compensates for something the model cannot yet do on its own, a more capable model needs less harness. Anthropic frames harness design as deciding "what belongs in that scaffolding and, as models improve, what you can take out," and observes that "every component in a harness encodes an assumption about what the model can't do on its own," an assumption that grows stale as the model improves.[1][7]

### Harness engineering

As agents moved into daily software work, tuning the harness became a discipline of its own. In February 2026 the engineer Mitchell Hashimoto described what he called "harness engineering," "the idea that anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again," for example by adding a tool that lets the agent verify its own work.[8] Anthropic's own experiments show the trade-off at the heart of the practice: a multi-agent harness can be "over 20x more expensive" to run than a single agent while producing output of visibly higher quality.[7]

## Evaluation harnesses

An evaluation harness is a framework that measures a model rather than deploying it. It takes a benchmark, which is a dataset paired with a task definition and a scoring rule, turns each example into a prompt, runs the model, extracts an answer from the output, computes a metric, and reports a comparable number.[16] The reason to package this as reusable software is reproducibility: if everyone runs the same benchmark through the same harness, their scores can be compared.

### The lm-evaluation-harness

The canonical example is EleutherAI's lm-evaluation-harness, described by its authors as "a framework for few-shot language model evaluation."[11] EleutherAI built it during its open replication of GPT-3, so that open models could be measured under the same few-shot methodology rather than trusting numbers copied from papers; the first release is cited as Gao et al., 2021.[11] The project has grown to cover "over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants," and it became the backend of Hugging Face's Open LLM Leaderboard, which used it to score thousands of open models on both its 2023 and its 2024 task suites.[12] The harness has been "used in hundreds of papers" and inside dozens of organizations; foundational open-model papers such as Pythia and GPT-NeoX-20B report their benchmark numbers with it.[12][14][15]

### How an evaluation harness works

Under the hood, a harness can score a model in more than one way, and the choice affects the result. For multiple-choice questions it can compare the probabilities the model assigns to each candidate answer, so that no text is generated and the chosen "answer" is simply the option the model finds most likely; the lm-evaluation-harness calls this a log-likelihood request. For open-ended tasks such as grade-school math or code, it instead lets the model generate text until a stop condition, then extracts the final answer and applies a metric like exact match; this is a generate-until request.[12] Seemingly minor decisions at this stage, the exact prompt wording, the number of in-context examples, and whether the answer is read from letter probabilities or from generated text, all move the number.

### Other evaluation frameworks

Several other frameworks fill the same niche. [OpenAI](/wiki/openai) released Evals in March 2023 alongside GPT-4 as "a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks," inviting the community to contribute tests.[20] Its later simple-evals library is deliberately lightweight and standardizes on a "zero-shot, chain-of-thought" setup, published, in OpenAI's words, "so we can be transparent about the accuracy numbers we're publishing."[21] Stanford's [HELM](/wiki/helm), short for Holistic Evaluation of Language Models, is both a large benchmark suite and a runnable evaluation framework, built for holistic, reproducible, and transparent measurement across many scenarios and metrics at once.[19]

## Reproducibility: the same model, different scores

Because so much depends on the harness, the same model run through two different harnesses can post very different scores on what is nominally the same benchmark. The best-documented case involves [MMLU](/wiki/mmlu), a multiple-choice knowledge test. In June 2023, as the first models were being added to the Open LLM Leaderboard, Hugging Face found that the published LLaMA scores did not match the ones the leaderboard was producing, and traced the gap to three different implementations of MMLU: the original code from the benchmark's authors, the version in Stanford's HELM, and the version in the lm-evaluation-harness.[13] For the 65-billion-parameter LLaMA model, the three gave 0.636, 0.637, and 0.488, a spread of about fifteen points on the same model answering the same questions.

| MMLU implementation | LLaMA-7B | LLaMA-13B | LLaMA-30B | LLaMA-65B |
| --- | --- | --- | --- | --- |
| Original (Hendrycks et al.) | 0.351 | 0.470 | 0.584 | 0.636 |
| HELM | 0.339 | 0.471 | 0.583 | 0.637 |
| lm-evaluation-harness | 0.342 | 0.377 | 0.457 | 0.488 |

The differences came from choices that sound trivial: whether the prompt includes a topic line, how the answer options are labeled, and, decisively, how the answer is read out. The original code compared the model's probabilities on just the four answer letters; HELM compared generated text to the answer text; the harness scored the probability of the full answer string, the letter followed by its text.[13] As Hugging Face concluded, "these different implementations of the same benchmark give widely different numbers and even change the ranking order of the models."[13] The episode is the standard cautionary tale that a benchmark score is close to meaningless without naming the harness that produced it.

### Agentic evaluation harnesses

For agents, evaluation harnesses have to do more than score text; they have to run the agent's actions and check the state of the world afterward. SWE-bench's harness takes the code patch an agent produces, applies it to a real repository inside a Docker container, and runs the project's test suite; an instance counts as resolved only when the tests tied to the issue now pass and the tests that already passed still pass.[16][17] The maintainers moved to "a fully containerized evaluation harness using Docker" in June 2024 specifically "for more reproducible evaluations."[18] Other agent benchmarks grade the outcome rather than the transcript in the same way: [Terminal-Bench](/wiki/terminal_bench) checks the final state of a container after a command-line task,[25] and the tool-use benchmark [tau-bench](/wiki/tau_bench) compares a database's final state against an annotated goal, adding a metric it calls pass^k for how reliably an agent repeats a success across independent trials.[26] The [GAIA](/wiki/gaia_benchmark) benchmark for general assistants grades short factual answers against a held-out key, and reported that human respondents answered 92% of its questions correctly against 15% for GPT-4 with plugins.[27] Because the score depends on the agent scaffold that produced the actions and the harness that graded them, an entry on one of these leaderboards is best read as a pairing of a model with a scaffold, not as a property of the model alone.

### Standardizing agent evaluation

The dependence of scores on the harness has prompted efforts to standardize it. The Holistic Agent Leaderboard, from a Princeton-led team, offers "a standardized evaluation harness that orchestrates parallel evaluations across hundreds of VMs," and used it to run 21,730 agent rollouts across nine models and nine benchmarks for about $40,000, arguing that without shared, standardized practice, agent results cannot be compared meaningfully across approaches.[28] The broader recommendation running through this work is to treat the harness as a controlled variable reported alongside the model, rather than an implementation detail hidden behind a single number.[29]

## Other meanings and disambiguation

Because the word is common, "harness" turns up in several contexts unrelated to the AI senses above.

The 2015 paper "Explaining and Harnessing Adversarial Examples," by [Ian Goodfellow](/wiki/ian_goodfellow), Jonathon Shlens, and Christian Szegedy, which introduced the Fast Gradient Sign Method, uses "harnessing" as an ordinary verb, meaning to put adversarial examples to work for adversarial training. It has nothing to do with a software harness.[30]

In robotics and manufacturing, a wire harness is a bound bundle of wires and cables; automating its assembly is a well-known open problem, because the material is flexible and every product is slightly different, and it is sometimes cited as a task that robots still struggle to perform. A safety harness is a body-worn assembly for fall protection. Finally, Harness, at harness.io, is a software-delivery company founded in 2017 by Jyoti Bansal; its continuous-integration and continuous-delivery platform shares the name by coincidence and is unrelated to the AI concept.[33]

## See also

- [AI agent](/wiki/agent)
- [Agentic AI](/wiki/agentic_ai)
- [Tool use](/wiki/tool_use)
- [Function calling](/wiki/function_calling)
- [Model Context Protocol](/wiki/model_context_protocol)
- [SWE-bench](/wiki/swe_bench)
- [Benchmark (AI)](/wiki/benchmark)
- [Model Evaluation](/wiki/model_evaluation)
- [METR](/wiki/metr)
- [Multi-agent system](/wiki/multi_agent_system)

## References

1. Martin, Lance. "Agent harness design: patterns for building agents." Anthropic (Claude Platform blog), April 2, 2026. https://claude.com/blog/harnessing-claudes-intelligence
2. Databricks. "What is an AI Agent Harness?" https://www.databricks.com/blog/ai-harness
3. Paniego, Sergio, and Aritra Roy Gosthipaty. "Harness, Scaffold, and the AI Agent Terms Worth Getting Right." Hugging Face, May 25, 2026. https://huggingface.co/blog/agent-glossary
4. Anthropic. "Building Effective Agents." December 19, 2024. https://www.anthropic.com/engineering/building-effective-agents
5. Schluntz, Erik. "Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet." Anthropic, January 6, 2025. https://www.anthropic.com/engineering/swe-bench-sonnet
6. Anthropic. "Effective harnesses for long-running agents." November 26, 2025. https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
7. Rajasekaran, Prithvi. "Harness design for long-running application development." Anthropic, March 24, 2026. https://www.anthropic.com/engineering/harness-design-long-running-apps
8. Hashimoto, Mitchell. "My AI Adoption Journey." February 5, 2026. https://mitchellh.com/writing/my-ai-adoption-journey
9. de Macedo, Sanderson Oliveira. "What makes a harness a harness: necessary and sufficient conditions for an agent harness." arXiv:2606.10106, June 8, 2026. https://arxiv.org/abs/2606.10106
10. Yang, John, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." arXiv:2405.15793, 2024 (NeurIPS 2024). https://arxiv.org/abs/2405.15793
11. Gao, Leo, et al. "A framework for few-shot language model evaluation." Zenodo, v0.4.0, December 2023. DOI 10.5281/zenodo.10256836 (original release v0.0.1, 2021, DOI 10.5281/zenodo.5371628). https://github.com/EleutherAI/lm-evaluation-harness
12. EleutherAI. "lm-evaluation-harness" (project README). https://github.com/EleutherAI/lm-evaluation-harness
13. Fourrier, Clémentine, Nathan Habib, Julien Launay, and Thomas Wolf. "What's going on with the Open LLM Leaderboard?" Hugging Face, June 23, 2023. https://huggingface.co/blog/open-llm-leaderboard-mmlu
14. Biderman, Stella, et al. "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling." arXiv:2304.01373, 2023. https://arxiv.org/abs/2304.01373
15. Black, Sid, et al. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model." arXiv:2204.06745, 2022. https://arxiv.org/abs/2204.06745
16. Jimenez, Carlos E., John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024. arXiv:2310.06770. https://arxiv.org/abs/2310.06770
17. SWE-bench. "Evaluation" (documentation). https://www.swebench.com/SWE-bench/guides/evaluation/
18. SWE-bench. Project README (containerized evaluation harness, June 27, 2024 note). https://github.com/SWE-bench/SWE-bench
19. Liang, Percy, et al. "Holistic Evaluation of Language Models." arXiv:2211.09110, 2022. Stanford Center for Research on Foundation Models. https://arxiv.org/abs/2211.09110 ; https://github.com/stanford-crfm/helm
20. OpenAI. "Evals: a framework for evaluating LLMs and LLM systems." March 2023. https://github.com/openai/evals
21. OpenAI. "simple-evals." https://github.com/openai/simple-evals
22. METR. "Guidelines for capability elicitation." March 15, 2024. https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/
23. METR. "Measuring the impact of post-training enhancements." March 15, 2024. https://metr.org/blog/2024-03-15-measuring-post-impact-enhancements/
24. METR. "Evaluating GPT-4o's autonomous capabilities" (agent scaffolding and elicitation). https://metr.org/evaluations/gpt-4o-report/
25. Merrill, Mike A., Alexander G. Shaw, et al. "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces." arXiv:2601.11868, 2026. https://arxiv.org/abs/2601.11868
26. Sierra Research. "τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv:2406.12045, 2024. https://arxiv.org/abs/2406.12045
27. Mialon, Grégoire, et al. "GAIA: a benchmark for General AI Assistants." arXiv:2311.12983, 2023. https://arxiv.org/abs/2311.12983
28. Kapoor, Sayash, Benedikt Stroebl, et al. "Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation." arXiv:2510.11977, October 13, 2025. https://arxiv.org/abs/2510.11977
29. Yao, Yilun, et al. "Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows." arXiv:2605.27922, May 27, 2026. https://arxiv.org/abs/2605.27922
30. Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and Harnessing Adversarial Examples." arXiv:1412.6572, 2014 (ICLR 2015). https://arxiv.org/abs/1412.6572
31. "Test harness." Wikipedia. https://en.wikipedia.org/wiki/Test_harness
32. TechTarget. "What is a test harness?" https://www.techtarget.com/searchsoftwarequality/definition/test-harness
33. Harness, Inc. "About us." https://www.harness.io/company/about-us
34. "harness (n.)." Online Etymology Dictionary. https://www.etymonline.com/word/harness