HumanEval

HumanEval is a benchmark for measuring whether a code-generating large language model can complete short Python functions so that they pass unit tests. It contains 164 hand-written tasks. Each task supplies a prompt built around a function signature and docstring, while the released dataset also includes a reference implementation and tests. Mark Chen, Jerry Tworek, and colleagues at OpenAI introduced the benchmark in the 2021 paper Evaluating Large Language Models Trained on Code, which presented OpenAI Codex.^[1]

HumanEval is an execution-based benchmark: it judges a completion by running code, not by comparing its text with a reference answer. Its best-known score is pass@k, the estimated probability that at least one of k sampled completions for a task passes every test. The pass@k concept predates HumanEval; the Codex paper credited the 2019 SPoC work and introduced an unbiased estimator designed for repeated samples.^[1][16]

The benchmark is influential, compact, and easy to reproduce, but its score has a narrow meaning. It evaluates functional correctness on a small set of standalone Python problems under a particular prompt, sampling, test, and execution protocol. It does not by itself measure repository-scale software engineering, security, maintainability, or performance in other programming languages. Later projects such as HumanEval+, MultiPL-E, HumanEval-X, and EvoEval modify or extend the original benchmark and should be reported under their own names.^[5][7][8][9]

Overview

HumanEval was designed for one form of AI code generation: synthesizing a standalone function from a natural-language description. The Codex paper describes the target as Python function generation from docstrings. The 164 tasks assess language comprehension, reasoning, algorithms, and simple mathematics; the authors wrote that some are comparable to simple software interview questions.^[1]

Scroll sideways for more →

The original paper reported an average of 7.7 tests per problem.^[1] Counts in later analyses can differ because some test programs loop over many inputs rather than expressing each input as a separate assert statement. EvalPlus, for example, counted an average of 9.6 original test inputs, or 7.3 after excluding four tasks that perform more than 100 randomized checks. Those figures use a different counting convention and do not contradict the paper's stated 7.7 average.^[5]

HumanEval is both a dataset and an evaluation workflow. The public repository contains the compressed JSON Lines dataset, an evaluator, and an execution helper. The repository's example workflow generates 200 completions for each of the 164 task IDs, producing 32,800 candidate programs, and then reports pass@1, pass@10, and pass@100.^[2] An evaluator can use a different number of samples, but a claimed pass@k result is not reproducible without stating that number and the rest of the decoding protocol.

Origin and Design Goals

The Codex project needed a way to determine whether generated code worked. Text-generation metrics are poorly suited to this question because many programs with different source text implement the same function, while superficially similar programs can behave differently. The paper compared execution-based evaluation with exact or fuzzy matching and showed that BLEU distributions for correct and incorrect Codex completions overlapped substantially on sampled HumanEval tasks.^[1]

HumanEval therefore uses functional correctness. A candidate is accepted when the assembled function executes and satisfies every check in the task's test program within the evaluator's time limit. This binary decision gives no credit for code that passes only some checks. It also does not require the candidate to resemble the canonical implementation. Any implementation that satisfies the tested behavior can pass.

The tasks were written rather than copied programmatically from public problem collections. This was an attempt to reduce overlap with Codex's training data, which included a large fraction of public Python code from GitHub. The paper was careful not to call hand-writing a guarantee of novelty. Its figure caption states that all problems were hand-written and not programmatically copied, while explicitly noting that this did not guarantee that a problem was novel.^[1]

That distinction matters. A newly written prompt can still express a common algorithm or resemble material already present in training data. Moreover, once HumanEval was publicly released, its prompts, solutions, and tests could be copied into repositories, tutorials, datasets, and later model-training corpora. Hand construction reduced one risk at the time of creation; it did not make the benchmark permanently contamination-free.

HumanEval also fit the development setting of the Codex models. The paper focused on standalone Python functions, and its evaluation prompt consisted of a header, function signature, and docstring. Completions were sampled until the model emitted one of several stop sequences intended to prevent it from continuing into unrelated top-level definitions or statements. The authors used nucleus sampling with top_p = 0.95 for sampling evaluations.^[1]

The dataset accompanied research that was closely connected to GitHub Copilot, but HumanEval should not be described as an evaluation of the whole Copilot product. The paper states that a distinct production version of Codex powered Copilot. HumanEval measured model completions under the paper's controlled function-synthesis protocol, not interactive editing, developer acceptance, integration with an editor, or end-to-end productivity.^[1]

Task Structure

The official dataset is stored as JSON Lines compressed with gzip. Its task identifiers run from HumanEval/0 through HumanEval/163. Each record contains five fields:^[2]

• task_id, the stable identifier for the problem
• prompt, Python source containing any imports, the function signature, and the docstring
• entry_point, the name of the function passed to the checker
• canonical_solution, the reference function body
• test, Python source that defines the task's checks

The prompt is the part intended for generation. The canonical solution and test program are released for evaluation and analysis, but they must not be placed in the model's generation context in a standard HumanEval run. Calling the tests "hidden" means that they are withheld from the model while it produces a completion. It does not mean that the tests remain secret from benchmark users.

The first task illustrates the format. Its entry point is has_close_elements. The prompt asks for a function that determines whether two numbers in a list are closer than a threshold and includes two doctest-style examples. The canonical solution and a separate check(candidate) function appear in the released record. Other tasks cover operations on strings, lists, numbers, and structured values, along with short algorithms and parsing routines.^[2]

Doctest-style examples are not present in every prompt, and they should not be confused with the held-out checker. Examples in a docstring are visible to the model and can constrain the intended behavior. The test field is appended only after generation. An evaluation that exposes the test program, reference solution, or outputs derived from them to the model is using a different protocol.

The public evaluator accepts a JSON Lines sample file in which each row contains a task_id and a completion. It groups completions by task, runs each against that task's checker, counts passed samples, and applies the pass@k estimator. It requires every problem to be attempted. The default requested values are 1, 10, and 100, but a value is returned only if every task has at least k samples.^[3]

The evaluator assembles executable source by concatenating:

1. the task prompt,
2. the generated completion,
3. the task's test program, and
4. a call of the form check(entry_point).

The official implementation executes each candidate in a separate process and uses a default timeout of three seconds in the evaluator. Results are recorded as passed, timed out, or failed.^[3][4] Environment details can still affect outcomes. Python version, operating system, available packages, process behavior, and resource pressure can turn an otherwise identical completion into a different result.

Functional Correctness

Functional correctness is more appropriate than reference-text matching for HumanEval because source code has many equivalent forms. A loop, a comprehension, a library call, and a recursive implementation may all satisfy the same specification. Conversely, changing one comparison operator can preserve most tokens while introducing a boundary-case error. HumanEval treats these cases according to observed execution behavior rather than token overlap.^[1]

Passing the tests is not the same as proving a program correct for every valid input. A test suite samples behaviors from the function's input domain. If the suite omits an edge case, an incorrect completion can pass. If a prompt leaves the valid input domain or exception behavior ambiguous, a reasonable completion can fail a checker that embodies a different interpretation. The benchmark's result is therefore best phrased as "passes the HumanEval tests under this protocol," not as an unconditional proof that the generated code is correct.

The canonical solution is also not a textual gold answer. It helps define expected behavior and can be used to generate expected outputs, but the evaluator does not require the generated completion to match it. This is a strength of execution-based evaluation. It also places considerable trust in the prompt, checker, reference implementation, and runtime as a combined specification.

HumanEval's binary task outcome avoids an especially misleading alternative: averaging the fraction of individual assertions passed by a program. A completion that handles common inputs but fails a critical case is still incorrect for that task. The original Codex evaluation counted it as a failure. This strict criterion makes pass@k easy to interpret, although it can hide which behavior caused the failure.

Pass@k

For one problem, suppose an evaluator generates n completions and c of them pass. The HumanEval estimator for a sampling budget k is:

1 - C(n - c, k) / C(n, k)

where C(a, b) is a binomial coefficient. The reported pass@k is the mean of this quantity over all tasks.^[1][3]

The expression asks: if k completions were selected uniformly without replacement from the n observed completions, what is the probability that the subset would include at least one of the c passing completions? Its complement is the probability that all k selected completions come from the n - c failures.

For example, if 40 of 200 sampled completions pass for one task, the estimator is:

• 0.20 for pass@1
• approximately 0.8987 for pass@10
• approximately 1.0 for pass@100

This does not mean that a deployed system will automatically identify the correct member of a 100-sample set. Pass@100 treats the tests as an oracle that can recognize a passing completion. A production system without tests or a reliable verifier faces a separate selection problem.

The metric name is often misunderstood:

• pass@1 estimates the probability that one sample passes under the stated sampling distribution.
• pass@10 estimates the probability that at least one member of a ten-sample budget passes.
• pass@100 estimates the same event for a 100-sample budget.

These are not top-k classification accuracies, rankings of candidates, or percentages of unit tests passed. They summarize success under repeated stochastic generation.

The pass@k idea was used before HumanEval. The Codex paper cited SPoC, a 2019 pseudocode-to-code system that searched candidate programs under a compilation budget and reported whether a problem was solved within that budget.^[16] HumanEval's specific methodological contribution was the combinatorial estimator. The paper generated n = 200 samples per task for k values no larger than 100 and showed that the simpler expression 1 - (1 - p_hat)^k, based on empirical pass@1, is biased.^[1]

The official evaluator implements the estimator in a numerically stable product form. When n - c < k, every possible subset of size k must include a correct completion, so it returns 1. Otherwise, it evaluates the complement term without constructing enormous binomial coefficients.^[3]

The condition n >= k is necessary but not sufficient for a strong comparison. Sampling only k candidates and marking whether any passes yields a valid observed success or failure for that run, but it is a high-variance way to estimate the underlying success probability. The original 200-sample design produces one estimate for several k values from the same candidate pool.

Greedy decoding creates another distinction. A deterministic greedy completion per task can be scored as the fraction of tasks passed, and papers sometimes label this value pass@1. It is not drawn from the same distribution as temperature-sampled pass@1. A comparison must say whether the reported number comes from greedy decoding or random sampling.

Sampling temperature changes the metric. The Codex paper found that lower temperature favored pass@1, while higher temperature could help large k by increasing diversity. For its 679-million-parameter Codex model, the best tested temperature was 0.2 for pass@1 and 0.8 for pass@100. A score without temperature, top_p, stop rules, and sample count is incomplete.^[1]

Original Codex Results

The paper's Table 1 reported the following HumanEval results for selected models under the authors' evaluated settings:^[1]

Scroll sideways for more →

The paper also evaluated Codex-S, obtained by further supervised fine-tuning on correctly implemented standalone functions. It reported 37.7% with one sample for Codex-S 12B. At temperature 0.8, generating 100 candidates and using the tests as an oracle found a passing sample for 77.5% of tasks; choosing only the candidate with the highest mean token log probability solved 44.5%.^[1]

Several numbers from the paper are easy to combine incorrectly. Its abstract says that Codex solved 28.8% with one sample and 70.2% with 100 samples per problem. Table 1 reports 72.31% pass@100 for Codex 12B under the best result from the temperatures evaluated for that table. The 77.5% figure refers to Codex-S, not the base Codex model. A historical summary should preserve these settings and model names instead of presenting the figures as interchangeable.^[1]

The finding was not merely that a larger language model performed better. The GPT family evaluated in the paper achieved near-zero performance, while models trained substantially on code performed better. This supported the paper's central conclusion that training distribution and code specialization mattered for executable program synthesis.^[1]

Those results are historical baselines, not a current leaderboard. Later papers have used different prompts, chat wrappers, sample counts, temperatures, tests, execution environments, and model versions. A percentage copied from a model card is comparable with the 2021 table only if those differences are reconciled.

Reproducible Evaluation

A reproducible HumanEval result requires more than a model name and one percentage. At minimum, an evaluation should record:

• the exact HumanEval dataset and evaluator revision,
• whether the original tests or an augmented suite such as HumanEval+ was used,
• the model checkpoint or API version and access date,
• the complete prompt template, including chat formatting and any system message,
• whether examples, imports, or function signatures were modified,
• stop sequences and maximum generation length,
• decoding method, temperature, top_p, and other sampling controls,
• number of generated samples n per task and each reported k,
• execution environment, dependency versions, timeout, and resource limits,
• task exclusions, generation failures, and retry policy,
• whether test data or reference solutions were screened out of the prompt and retrieval context, and
• what contamination checks were applied to the model's training data, when such data are available.

Prompt wrappers matter because HumanEval was created for code completion. Chat and instruction-tuned models often require the original prompt to be embedded in a conversation template, and their outputs may contain Markdown fences, explanations, or a repeated function signature. Stripping or repairing such output can change the result. The transformation should be specified and applied consistently.

Sample allocation must also be uniform or documented. The official evaluator refuses to calculate a requested pass@k unless each task has at least k candidates.^[3] If an API failure leaves fewer samples for some tasks, silently averaging the remaining results changes the target quantity. A defensible run either restores the intended sample count or reports the missing-data rule.

Test execution should distinguish model failure from infrastructure failure. Syntax errors and failing assertions are candidate failures. A machine running out of memory, a missing runtime dependency, or an incompatible platform may be an evaluation failure. The official repository notes that memory exhaustion can cause correct programs to fail and recommends rerunning after freeing memory.^[2]

HumanEval's small size makes evaluation convenient but also makes individual tasks visible in aggregate scores. In a deterministic one-completion evaluation, one task is approximately 0.61 percentage points of the 164-task total. Reporting extra decimal places does not remove prompt sensitivity, execution variation, contamination uncertainty, or sampling variance.

For high pass@1 values, pass@10 and pass@100 can approach the ceiling rapidly. These metrics then provide little separation, even if the underlying per-sample probabilities differ. Conversely, a large gap between pass@1 and pass@100 shows that correct programs appear somewhere in the sampled distribution, not that the model can select them without an oracle.

Common Scoring Errors

Several aggregation mistakes can produce a number labeled pass@k that does not match the HumanEval definition.

Pooling all completions across tasks. The estimator is calculated separately for every problem and then averaged. Pooling all passing and failing completions first would give easier tasks, or tasks with more samples, disproportionate influence. A task with 200 passing completions and a task with none should contribute one task-level value each, not be treated as one combined collection of 400 programs.^[1][3]

Using different n values without recording them. The estimator can accept a different sample count for each task, but each task still needs at least k samples. Unequal counts may be unavoidable after generation failures. They should be retained explicitly in the calculation and disclosed, not replaced with a global assumed n.^[3]

Treating pass@100 as an interactive 100-step process. The standard metric draws a subset from independently generated candidates under a fixed prompt and decoding setup. A system that examines test failures and revises its next attempt uses additional information. That can be a valuable repair evaluation, but it is not the original pass@100 protocol.

Selecting a temperature on the reported test set without disclosure. The Codex paper evaluated several temperatures and reported the best result for each k in some tables.^[1] Repeatedly tuning prompts or decoding settings against the same 164 tasks can itself overfit the benchmark. A study should distinguish prespecified settings from test-set selection and, where possible, validate choices on separate development tasks.

Confusing percentage points with relative percent. If an augmented test suite changes a score from 80% to 70%, the absolute drop is 10 percentage points and the relative decrease is 12.5%. EvalPlus reported reductions in percentage points for its comparisons.^[5]

Rounding before averaging. Task-level estimates should be averaged at full precision, with rounding applied only to the final reported result. Rounding every task first can shift the aggregate, particularly for higher k.

Mixing benchmark variants. Original HumanEval, corrected copies, HumanEval+, and harness-specific adaptations can return different outcomes for the same completion. A table that labels all of them simply "HumanEval" conceals a substantive experimental difference.

These errors are avoidable when the evaluation publishes its configuration and, ideally, the per-task sample counts and outcomes. A single aggregate remains useful, but the underlying records make it possible to audit exclusions, reproduce the estimator, and separate model behavior from harness failures.

Strengths

HumanEval has several enduring strengths:

Executable judgment. It accepts semantically different implementations when they satisfy the checks. This is more meaningful for program synthesis than measuring similarity to one reference solution.^[1]

Simple task contract. A prompt leads to a completion, and a checker returns a binary outcome. The structure makes the benchmark easy to integrate into model-development and evaluation systems.

Multiple-sample analysis. The unbiased pass@k estimator separates single-sample success from the availability of a solution in a larger stochastic candidate set.^[1][3]

Hand-written test set. Creating new tasks reduced direct copying from known public problem collections at the time Codex was trained, even though it could not guarantee semantic novelty or prevent later contamination.^[1]

Public artifacts. The prompts, canonical solutions, tests, evaluator, and execution helper are available under an MIT license, allowing researchers to inspect the complete evaluation rather than relying on a private scoring service.^[2]

These strengths explain why HumanEval became a common baseline. They do not make it a comprehensive measurement of programming ability. Its clearest use is as a controlled test of short Python function synthesis, especially when a study needs continuity with earlier code-model research.

Limitations

Limited Test Coverage

Passing a finite suite is only evidence about the inputs it covers. The original paper's 7.7-test average left many opportunities for false positives. An implementation may handle the visible pattern but fail on empty collections, boundary values, repeated elements, unusual numeric ranges, or performance-heavy inputs.

EvalPlus demonstrated this problem empirically. Its authors expanded HumanEval's tests by about 80 times and found that the added cases reduced pass@k by as much as 19.3 to 28.9 percentage points across the studied settings. They also found ranking changes among models, showing that incomplete tests can affect not only absolute scores but comparative conclusions.^[5]

Ambiguous Specifications

A docstring may omit an input precondition, tie-breaking rule, exception policy, numeric tolerance, or performance requirement. The test program then acts as an implicit specification. A candidate that chooses a plausible but different interpretation can fail, while a candidate tailored to the checker can pass.

This is not fully solved by adding more tests. More tests improve coverage of one interpretation, but they do not automatically make the natural-language requirement unambiguous. EvalPlus addressed part of this issue by adding hand-crafted input contracts to 83 of 164 HumanEval+ tasks.^[5]

Errors In Reference Implementations

The original benchmark artifacts were not error-free. The EvalPlus study reported 18 defects affecting 11% of HumanEval problems: five unhandled edge cases, ten logic errors, and three performance problems in original reference implementations. Its authors reimplemented and tested the ground truths used for HumanEval+.^[5]

This finding does not mean that every original HumanEval score is invalid, because a flaw matters only when it changes a tested outcome. It does mean that the original and corrected artifacts are not interchangeable. Evaluators should identify the exact dataset revision and avoid assuming that a canonical solution is infallible.

Narrow Programming Scope

HumanEval evaluates 164 standalone Python functions. It does not directly test:

• navigating an existing repository,
• coordinating changes across files,
• understanding a build system or dependency graph,
• resolving an issue report,
• writing migrations, user interfaces, or deployment configuration,
• interacting with external services,
• maintaining long-lived state,
• reviewing code written by another developer, or
• explaining and negotiating ambiguous product requirements.

It also provides little direct evidence about code in another programming language. A model can have uneven training coverage, tokenization, library knowledge, and compiler feedback across languages. Multilingual variants exist because a Python result cannot simply be assumed to transfer.

Weak Coverage Of Nonfunctional Quality

The main outcome is functional acceptance under tests and a timeout. It does not systematically score readability, maintainability, documentation, security, privacy, energy use, algorithmic complexity, dependency quality, or compatibility with a larger system. A concise, robust implementation and a brittle implementation can receive the same result if both pass.

Timeouts provide a limited efficiency constraint, but they are not a calibrated performance benchmark. Runtime depends on the evaluator's hardware and operating system, while many HumanEval inputs are too small to distinguish algorithmic complexity reliably.

Oracle Selection

At k > 1, pass@k assumes that a passing member of the candidate set can be recognized by the benchmark tests. This is appropriate for measuring candidate availability. It can overstate the performance of a deployed generator that lacks those tests.

The Codex paper separated oracle selection from feasible reranking. Codex-S 12B reached 77.5% when the tests selected a passing result from 100 samples, but the highest-mean-log-probability candidate passed 44.5%. The gap is evidence about selection difficulty, not a contradiction between metrics.^[1]

Public-Benchmark Contamination

HumanEval's complete artifacts have been public since 2021. Later model-training corpora can include exact problems, paraphrases, translations, solutions, benchmark-specific explanations, or synthetic data derived from them. A high score can then reflect some mixture of general code ability and exposure to the evaluation.

Yang and colleagues tested contamination detectors on rephrased benchmark samples. Their LLM-based method identified overlap corresponding to 8% to 18% of HumanEval in sampled subsets of RedPajama-Data-1T and StarCoder-Data. They also showed that training CodeLlama models on rephrased HumanEval material could sharply increase benchmark scores while evading ordinary n-gram matching.^[6]

These results establish a risk, not proof that every later model memorized HumanEval. Closed training data often prevent a definitive audit. Responsible reporting should state what is known about data cutoffs and decontamination, avoid interpreting a public benchmark as a sealed exam, and use fresh or time-filtered tasks when contamination sensitivity is central.

Execution Safety

HumanEval evaluates model-generated Python by executing it. Generated code is untrusted even when the prompt is benign. It can accidentally consume resources, access files, spawn processes, or perform destructive actions.

The official repository strongly warns users not to run the harness outside a robust security sandbox. Its reliability_guard disables a number of functions, but the code itself states that this guard is not a security sandbox. The original paper used gVisor, network firewall rules, and other isolation controls in its research environment.^[1][2][4]

Running the evaluator in an ordinary development environment is therefore unsafe. A proper setup should isolate the process, filesystem, network, credentials, and host resources and should assume that both generated code and third-party evaluation artifacts can be hostile.

HumanEval+ and EvalPlus

HumanEval+ is an augmented evaluation built from the same 164 task concepts, not a newer name for the original benchmark. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang introduced it with EvalPlus at NeurIPS 2023.^[5]

EvalPlus combines language-model-generated seed inputs with type-aware mutation. For each task in the reported construction, the authors used about 30 seed inputs generated through three prompts and then generated 1,000 additional inputs under a one-hour mutation budget. They used differential testing against corrected reference implementations and program contracts to filter invalid inputs.^[5]

The paper reported these test-suite statistics:

Scroll sideways for more →

The original row includes four tasks with more than 100 randomized checks; excluding those changes the original average to 7.3 and maximum to 26. HumanEval+-mini was produced through test-suite reduction to retain nearly the same effectiveness on the studied model samples at much lower execution cost.^[5]

Across 26 evaluated models, the expanded tests exposed previously undetected wrong programs. The largest observed reductions were 19.3 to 28.9 percentage points for different k values. The study also found cases in which model rankings changed. This is direct evidence that a HumanEval score and a HumanEval+ score cannot be placed in one column without identifying the test suite.^[5]

HumanEval+ improves test rigor, but it does not remove every limitation of HumanEval. The task set remains small, Python-based, public, and centered on standalone function synthesis. Its tests also encode the corrected specifications chosen by the EvalPlus authors. The appropriate interpretation is stronger behavioral checking on the HumanEval task family, not a complete evaluation of software engineering.

Contamination-Resistant and Evolved Evaluations

Two later research directions respond to different weaknesses of static HumanEval.

EvoEval evolves HumanEval-style tasks into targeted variants, including difficult, creative, subtle, combined, and tool-using problems. Its paper reported 828 problems across seven benchmark sets and evaluated 51 models. Average performance fell 39.4% relative to HumanEval, with model-level decreases ranging from 19.6% to 47.7%, and rankings changed substantially. The authors interpreted this as evidence of limited coverage and potential overfitting on established benchmarks.^[7]

LiveCodeBench attaches release dates to programming-contest problems and permits evaluation only on tasks published after a model's stated cutoff. Its introductory paper collected problems over time from LeetCode, AtCoder, and Codeforces and evaluated generation, self-repair, code execution, and test-output prediction. The study found one cluster of models that performed well on both HumanEval+ and LiveCodeBench and another, largely fine-tuned cluster, that performed well on HumanEval+ but not on LiveCodeBench, which the authors described as potential HumanEval overfitting.^[11]

These approaches answer different questions. Evolved tasks test whether performance survives controlled changes to familiar problem structures. Time-filtered tasks reduce the chance that an exact problem predates a model's training cutoff. Neither makes HumanEval obsolete as a historical baseline, but both show why a single static score is insufficient for strong claims about current coding ability.

Multilingual Variants

Several projects extend the HumanEval task family beyond English-to-Python generation. They are separate datasets with distinct construction choices.

MultiPL-E

MultiPL-E is a translation system and evaluation framework for code-generation benchmarks. Federico Cassano and colleagues used it to extend HumanEval and MBPP from Python to 18 additional programming languages, producing parallel evaluations across 19 languages.^[8]

The system translates function signatures, values, unit tests, doctest syntax, type information, and Python-specific terminology. Translation is not always one-to-one. The paper excluded three HumanEval tasks with helper functions in their prompts, modified two tasks that used randomized testing, and could not compile as many as five tasks for some typed languages because their types had no suitable translation.^[8]

MultiPL-E therefore supports cross-language comparisons, but its translated tasks are not identical byte-for-byte realizations of the Python benchmark. Language-specific type systems, equality behavior, rounding, standard libraries, prompt conventions, compilers, and stop sequences affect the evaluation.

HumanEval-X

HumanEval-X was introduced with CodeGeeX. Its authors manually rewrote each of the 164 HumanEval problems for C++, Java, JavaScript, and Go, alongside Python. The result is 820 problem-solution pairs across five languages. Each pair contains a declaration, docstring, prompt, canonical solution, and test program and can support code generation or translation.^[9]

Manual rewriting allowed the authors to adapt naming conventions, types, truth values, equality, rounding, and tests to each language. That is a different construction strategy from MultiPL-E's compiler-based approach. Results from the two extensions should not be combined merely because both derive from HumanEval.

HumanEval-XL

HumanEval-XL targets both natural-language and programming-language diversity. Its paper describes connections between 23 natural languages and 12 programming languages, totaling 22,080 prompts with an average of 8.33 test cases.^[10] It is designed to study whether models can generate code from descriptions in languages other than English, a dimension not measured by the original HumanEval.

The names HumanEval+, HumanEval-X, and HumanEval-XL are similar but refer to different interventions: more tests, more programming languages, and cross-lingual natural-language prompts, respectively.

Comparison With Other Code Benchmarks

HumanEval is one point in a larger benchmark design space:

Scroll sideways for more →

MBPP was created by crowd workers with basic Python knowledge. Each of its 974 tasks includes a natural-language description, a self-contained function, a reference solution, and three tests. Its paper also discusses a sanitized subset because some original descriptions or tests were ambiguous.^[14] MBPP is larger than HumanEval but shares its focus on short Python synthesis.

APPS collected 10,000 problems from open-access coding sites, split evenly into training and test sets. It spans introductory, interview, and competition difficulty. Many tasks require complete programs that read input and print output rather than completing a supplied function.^[15] It is therefore broader and often harder, but its public-source construction creates its own contamination considerations.

BigCodeBench was designed for richer library and function use. Its introductory paper reports 1,140 Python tasks that invoke functions from 139 libraries across seven domains, with an average of 5.6 test cases and 99% average branch coverage. It evaluates both structured completion prompts and a natural-language instruction variant.^[12] This makes it more informative about API composition than HumanEval, although it remains a function-oriented Python benchmark.

SWE-bench uses actual issue reports and repository snapshots. Its 2,294 original tasks come from 12 Python repositories and may require coordinated edits across multiple functions, classes, or files.^[13] A HumanEval completion can be generated from a short prompt in isolation; a SWE-bench system must inspect a codebase, produce a patch, and satisfy repository tests. Scores from the two benchmarks measure different capabilities.

No benchmark in the table provides a universal measure of "coding ability." Task source, prompt format, language, tool access, context length, test strength, contamination risk, and execution protocol determine what a result means.

Interpreting Results

A defensible HumanEval claim should answer three questions.

What Was Generated?

Specify the exact model, checkpoint, prompt, chat wrapper, and output processing. If the evaluator removed Markdown fences, extracted only a function body, retried malformed responses, or allowed the model to repair failed code, those are material parts of the system.

HumanEval traditionally supplies the prompt once and evaluates the resulting completion. An agent that can run tests, inspect failures, edit files, or call tools is being evaluated under an interactive scaffold. Its result may be useful, but it should not be compared as if it were plain completion.

What Was Counted?

State whether the tests were original HumanEval, HumanEval+, or another corrected suite. State n, k, greedy versus sampled decoding, temperature, and whether every task received the same number of attempts. Distinguish a sample-based pass@1 estimate from the fraction solved by one deterministic output.

Avoid phrases such as "accuracy on HumanEval" when they obscure the protocol. "HumanEval pass@1 with 200 samples per task at temperature 0.2" is more informative than "HumanEval accuracy." For an augmented suite, include the variant name in the metric label.

What Can Be Concluded?

A strong HumanEval result supports a bounded conclusion: under the reported protocol, the system frequently generates short Python functions that satisfy these tests. It does not alone establish:

• general competence across programming languages,
• correct behavior on unseen repository work,
• freedom from memorization or contamination,
• secure or maintainable output,
• efficient selection of a correct candidate without tests,
• effective collaboration with developers, or
• replacement of professional software engineering.

Claims about these broader properties require benchmarks or studies that directly measure them.

Comparisons across papers should be treated cautiously when any protocol element differs. Even a nominally identical pass@1 can refer to greedy decoding in one report, sampled decoding in another, original tests in one, HumanEval+ in another, or a chat prompt rather than raw completion. Re-running models in one harness is generally more informative than copying scores from unrelated tables.

Continuing Role

HumanEval remains useful for historical continuity, evaluator debugging, and controlled studies of function synthesis and sampling. Its small size allows researchers to inspect individual failures and run many candidate generations. The open artifacts also make it suitable for studying test quality, contamination, multilingual translation, reranking, and execution-based evaluation.

Its limitations should shape its role. For modern model comparisons, HumanEval is most informative as one component of a suite that includes stronger tests, fresh tasks, language diversity, library use, and repository-level work. A model can perform well on HumanEval and still fail substantially on evolved, time-filtered, tool-using, or multi-file tasks.^{[5][7][11][12][13]}

HumanEval's lasting contribution is therefore not a timeless leaderboard. It helped normalize execution-based evaluation of generated code and provided a clear repeated-sampling protocol. Used with explicit settings and careful scope, it remains a valuable benchmark. Used as a context-free percentage or a comprehensive measure of software engineering, it invites conclusions that its design cannot support.

References

1. ^Chen, M., Tworek, J., Jun, H., et al. (2021). "Evaluating Large Language Models Trained on Code." arXiv:2107.03374. arxiv.org/...2107.03374
2. ^OpenAI. "HumanEval: Hand-Written Evaluation Set." Official dataset and evaluation repository. github.com/...human-eval
3. ^OpenAI. human_eval/evaluation.py. Official HumanEval pass@k and evaluation implementation. github.com/...evaluation.py
4. ^OpenAI. human_eval/execution.py. Official HumanEval execution helper and sandbox warning. github.com/...execution.py
5. ^Liu, J., Xia, C. S., Wang, Y., and Zhang, L. (2023). "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation." Advances in Neural Information Processing Systems 36. proceedings.neurips.cc/...8686-Abstract-Conference
6. ^Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I. (2023). "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples." arXiv:2311.04850. arxiv.org/...2311.04850
7. ^Xia, C. S., Deng, Y., and Zhang, L. (2024). "Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM." arXiv:2403.19114. arxiv.org/...2403.19114
8. ^Cassano, F., Gouwar, J., Nguyen, D., et al. (2022). "MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation." arXiv:2208.08227. arxiv.org/...2208.08227
9. ^Zheng, Q., Xia, X., Zou, X., et al. (2023). "CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X." Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. arxiv.org/...2303.17568
10. ^Peng, Q., Chai, Y., and Li, X. (2024). "HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization." LREC-COLING 2024. arxiv.org/...2402.16694
11. ^Jain, N., Han, K., Gu, A., et al. (2024). "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code." arXiv:2403.07974. arxiv.org/...2403.07974
12. ^Zhuo, T. Y., Vu, M. C., Chim, J., et al. (2025). "BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions." International Conference on Learning Representations. arxiv.org/...2406.15877
13. ^Jimenez, C. E., Yang, J., Wettig, A., et al. (2024). "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" International Conference on Learning Representations. arxiv.org/...2310.06770
14. ^Austin, J., Odena, A., Nye, M., et al. (2021). "Program Synthesis with Large Language Models." arXiv:2108.07732. arxiv.org/...2108.07732
15. ^Hendrycks, D., Basart, S., Kadavath, S., et al. (2021). "Measuring Coding Challenge Competence With APPS." Advances in Neural Information Processing Systems 34, Datasets and Benchmarks Track. arxiv.org/...2105.09938
16. ^Kulal, S., Pasupat, P., Chandra, K., Lee, M., Padon, O., Aiken, A., and Liang, P. (2019). "SPoC: Search-based Pseudocode to Code." Advances in Neural Information Processing Systems 32. proceedings.neurips.cc/...ca44cc69ecf6f6b-Abstract