LM Evaluation Harness

LM Evaluation Harness (the Language Model Evaluation Harness, often abbreviated lm-eval or written lm-evaluation-harness) is an open-source software framework for measuring the performance of large language models on a wide range of benchmarks. Maintained by EleutherAI, it standardizes the steps between a benchmark and a model, formatting each task's prompts, running the model, extracting and scoring the answers, and reporting metrics, so that results are reproducible and comparable across models and across research groups. It is the most widely used open evaluation harness in language-model research, and it serves as the backend of Hugging Face's Open LLM Leaderboard.^[1][4]

The project describes itself as "a framework for few-shot evaluation of language models," and it covers "over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants."^[1] Its canonical citation is Gao et al., first released in 2021 and substantially rewritten in December 2023; the latest release at the time of writing is version 0.4.12, from May 2026.^[1][2]

History

EleutherAI, a grassroots research collective formed in 2020, built the harness during its effort to train and release open replications of GPT-3, including the GPT-Neo, GPT-J, and GPT-NeoX models. The group needed a way to benchmark its own models under the same few-shot methodology that OpenAI had reported for GPT-3, rather than trusting numbers copied from different papers that used different prompting.^[1] The first citable release, version 0.0.1, appeared in 2021 and is the "Gao et al., 2021" reference that recurs throughout the open-model literature.^[2]

In December 2023 the project shipped version 0.4.0, a large rewrite known as the "big refactor," which introduced a task-configuration format based on YAML files and a unified request interface for models. The current citation points to that release, "A framework for few-shot language model evaluation," archived on Zenodo.^[2] Development has continued steadily since; version 0.4.12 was released in May 2026.^[1]

How it works

An evaluation run turns each example in a benchmark into a prompt, optionally prepends a fixed number of in-context examples (the few-shot setting), runs the model, and scores the output. The harness asks a model to answer requests of three kinds, and the choice of request determines how a task is measured:^[3]

• loglikelihood: given an input and a candidate continuation, return the log-probability the model assigns to that continuation. Multiple-choice tasks use this to rank the candidate answers by probability, with no text generated.
• loglikelihood_rolling: return the log-probability of an entire string, used to compute perplexity over a corpus.
• generate_until: sample text until a stop condition, used for open-ended tasks such as math or code, after which an answer is extracted from the output and scored with a metric such as exact match.

The harness can drive many model backends, among them Hugging Face Transformers models, high-throughput inference servers such as vLLM, and commercial models served behind OpenAI-compatible APIs. To support reproducibility, it versions its task definitions and prompts and can log the exact inputs and outputs for every sample.^[1]

Adoption

The harness became shared infrastructure for the open-model community. It is the backend of Hugging Face's Open LLM Leaderboard, which used it to score thousands of open-weight models: first on a 2023 suite of six tasks (ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, and GSM8K), then, after a June 2024 relaunch, on a harder suite (IFEval, BBH, MATH, GPQA, MuSR, and MMLU-Pro) before the leaderboard stopped taking submissions in 2025.^[4][7] By EleutherAI's own account the harness "has been used in hundreds of papers" and "is used internally by dozens of organizations including NVIDIA, Cohere, BigScience, BigCode, Nous Research, and Mosaic ML."^[1] Foundational open-model papers, including Pythia and GPT-NeoX-20B, report their benchmark numbers with it.^[5][6]

Reproducibility and why the harness matters

Because a benchmark score depends on details that the harness fixes, the prompt wording, the number of shots, and above all how the answer is read out, the same model can post different numbers under different harnesses. The best-known illustration came in June 2023, when Hugging Face traced a discrepancy in reported MMLU scores to three different implementations of that benchmark: the original code, HELM, and the LM Evaluation Harness each scored the same LLaMA model differently, in one case by about fifteen points, because they read the model's answer in different ways.^[4] The episode is a standard argument that a benchmark result is only meaningful when the harness that produced it is named. The community later adjusted the harness's MMLU implementation, and the wider lesson is discussed in the article on evaluation harnesses.

Related tools

Several other frameworks occupy the same space. Stanford's HELM is a comparable framework that emphasizes reporting many metrics at once. OpenAI's Evals is an open framework and benchmark registry the company uses in its own model reporting, and Hugging Face's lighteval, influenced by both the LM Evaluation Harness and HELM, helped run the second version of the Open LLM Leaderboard.^[8][9][10] These frameworks differ in their default methodology, for example few-shot log-likelihood scoring versus zero-shot chain-of-thought generation, which is one reason their scores are not directly interchangeable.

See also

• Harness (AI)
• Benchmark (AI)
• Model Evaluation
• EleutherAI
• MMLU
• HELM (Holistic Evaluation of Language Models)

References

1. EleutherAI. "lm-evaluation-harness" (project README). GitHub. https://github.com/EleutherAI/lm-evaluation-harness ↩
2. Gao, Leo, et al. "A framework for few-shot language model evaluation." Zenodo, version 0.4.0, December 2023. DOI 10.5281/zenodo.10256836 (original release version 0.0.1, 2021, DOI 10.5281/zenodo.5371628). https://github.com/EleutherAI/lm-evaluation-harness ↩
3. EleutherAI. "Model guide" (request types: loglikelihood, loglikelihood_rolling, generate_until). lm-evaluation-harness documentation. https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md ↩
4. Fourrier, Clémentine, Nathan Habib, Julien Launay, and Thomas Wolf. "What's going on with the Open LLM Leaderboard?" Hugging Face, June 23, 2023. https://huggingface.co/blog/open-llm-leaderboard-mmlu ↩
5. Biderman, Stella, et al. "Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling." arXiv:2304.01373, 2023. https://arxiv.org/abs/2304.01373 ↩
6. Black, Sid, et al. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model." arXiv:2204.06745, 2022. https://arxiv.org/abs/2204.06745 ↩
7. Hugging Face. "Open LLM Leaderboard" (about and archive documentation). https://huggingface.co/docs/leaderboards/en/open_llm_leaderboard/about ↩
8. Liang, Percy, et al. "Holistic Evaluation of Language Models." arXiv:2211.09110, 2022. Stanford Center for Research on Foundation Models. https://arxiv.org/abs/2211.09110 ↩
9. OpenAI. "Evals: a framework for evaluating LLMs and LLM systems." https://github.com/openai/evals ↩
10. Hugging Face. "lighteval." https://github.com/huggingface/lighteval ↩