LLM-as-a-Verifier

LLM-as-a-Verifier is a probabilistic verification framework and open-source Python package for scoring and selecting large language model agent trajectories. It was introduced in a July 2026 arXiv preprint by researchers affiliated with Stanford University, the University of California, Berkeley, and NVIDIA Research. The method asks a language or vision-language model to compare candidate trajectories, then derives a continuous reward from the probabilities assigned to ordered score tokens instead of retaining only one generated score.^[1][2]

The framework can use those rewards to select one trajectory from a best-of-N candidate pool, estimate progress along a trajectory, or supply a dense reward in reinforcement learning. These uses add inference work and do not change the weights of the verifier model. The authors therefore call the method training-free, but that description does not mean compute-free or cost-free.^[1][2]

Evidence for the framework remained preliminary as of August 19, 2026. The principal results came from the authors' preprint and repositories, not an independent replication. The paper's benchmark figures describe systems composed of a candidate model, verifier model, agent harness, candidate-pool size, and benchmark snapshot. They are not new standalone scores for any one foundation model.^[1][6][7][8]

Publication and software

Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, and Azalia Mirhoseini submitted the paper to arXiv on July 6, 2026 and revised it on July 7. Version 2 lists Stanford, UC Berkeley, and NVIDIA Research affiliations. It was an arXiv preprint rather than a peer-reviewed conference or journal paper at that date.^[1]

The authors released the llm-verifier package on PyPI, with version 0.1.0 uploaded on July 7 and version 0.2.0 on August 14. The repository and package use the MIT License. Public interfaces include pairwise comparison, candidate selection, completed-trajectory scoring, and incremental progress tracking. The repository also contains stored benchmark trajectories and scripts intended to reproduce its reported selections.^[2][3][4]

The name refers to this paper and software, not to every use of an LLM for checking an answer. It belongs to the broader LLM-as-a-judge and verifier literature, but defines a particular score estimator and a particular candidate-ranking procedure.^[1][14]

Method

For one task, the framework presents a verifier with two candidate trajectories and an evaluation criterion. The prompt asks for a score for each trajectory. Explanatory material uses a 1-to-20 scale, while the implementation uses the ordered letters A through T to make extraction of token probabilities more reliable. For every score token, the system multiplies its mapped scalar value by the probability that the verifier assigned to that token, then sums across the distribution. The result is an expected score rather than the single most likely score token.^[1][2]

The score is averaged along three axes:^[1]

Scroll sideways for more →

For code-agent trajectories, the paper used Specification, Output, and Errors as separate criteria. After averaging, it normalized each trajectory's score to a zero-to-one range. A Bradley-Terry logistic function converted the difference between two scores into a soft preference probability. This probability, rather than a binary win, was passed to the ranking procedure.^[1]

The main scaling ablation used Gemini 2.5 Flash as the verifier and 200 randomly sampled trajectories from Terminal-Bench 2.0 across multiple agent harnesses. Pairwise verification accuracy increased from 73.1% at score granularity G=1 to 77.5% at G=20, from 74.7% with one evaluation to about 77.5% with 16 evaluations, and from between 75.2% and 76.4% for any individual code criterion to 78.3% for the three-criterion ensemble. These percentages measure how often a known successful trajectory was scored above a failed trajectory. They do not measure how many benchmark tasks an agent solved.^[1]

The difference between fewer ties and greater accuracy is visible in the paper's query-optimize case study. Across 100 repeated comparisons, a discrete five-level judge tied 88 times. Taking the expectation over the same five-level distribution removed all ties but ordered the pair correctly 69 times. Expanding the continuous scale to 20 tokens ordered it correctly 77 times and incorrectly 23 times. A continuous score can therefore force a ranking without making that ranking correct.^[1]

Probabilistic Pivot Tournament

Probabilistic Pivot Tournament, or PPT, ranks N candidates without scheduling every possible pair. It first places candidates on a random ring and scores adjacent pairs. Every candidate appears once in the A position and once in the B position, which is intended to cancel position bias in expectation. The candidates with the highest ring scores become k pivots. The system then compares each non-pivot with every pivot and compares the pivots with one another.^[1]

Soft win probabilities are added for each candidate and divided by that candidate's number of comparisons. The candidate with the largest normalized win mass is returned. Before removing any repeated ring edge, the schedule contains N + k(N-k) + k(k-1)/2 pair comparisons, so its leading growth is proportional to Nk when k is much smaller than N. A full round robin grows with N squared.^[1]

In one author-run ranking study, the pool contained 20 Terminus-2 trajectories for each of 89 Terminal-Bench tasks. PPT with three pivots selected a successful trajectory for 66.17% of tasks using 4,723 queried pairs. A full round robin reached 67.42% using 13,111 pairs, while the sample-average Pass@1 of that pool was 52.64%. The paper also compared PPT with V1, a March 2026 pairwise self-verification framework. These results concern the success of selection over that fixed pool, not the earlier pairwise-verification ablation.^[1][17]

Interpreting the metrics

Several quantities that appear together in the paper answer different questions:^[1][6][9]

Scroll sideways for more →

Best-of-N selection is a form of test-time compute: generating more candidates can expose solutions that a single sample misses, but the selector can recover only part of the oracle headroom. The paper also reports a pooled Terminal-Bench oracle value of 98.9% across trajectories from a larger leaderboard pool. That number says that at least one stored trajectory solved nearly every task; the proposed verifier did not achieve 98.9%.^[1]

Results reported in the paper

The July paper applied the same general scoring method in four settings, but the candidate models and metrics differed. The first three rows below report task success after selecting one candidate. The robotics row reports preference accuracy on pairs of videos and is not directly comparable with those task-success rates.^{[1][6][9][12][13]}

Scroll sideways for more →

The Terminal-Bench comparison was internally matched on the same five-candidate pool: the verifier raised the submitted success rate by 3.4 percentage points over the pool's 83.1% sample average but remained 5.6 points below the oracle Pass@5. The selected 86.5% is a result for GPT-5.5 candidates, Capy, Gemini 2.5 Flash verification, and the paper's benchmark configuration together. It is neither GPT-5.5 Pass@1 nor Gemini 2.5 Flash Pass@1.^[1]

The SWE-bench experiment used a heterogeneous pool rather than repeated samples from one model. The pool average was the mean success rate across one trajectory from each of three candidate models, and the oracle meant that any of those three solved the issue. The verifier's 78.2% result was 2.1 points above the 76.1% pool average and 6.2 points below the 84.4% oracle.^[1]

MedAgentBench contains 300 physician-written tasks in a simulated, FHIR-compatible electronic health-record environment. The verifier experiment used five trajectories from one candidate model and reported a 3.1-point increase over the pool average. This is a benchmark result in a simulation, not evidence that verifier-selected agents are safe for clinical deployment.^[1][12]

For RoboRewardBench, the authors created pairs of rollouts following the same instruction and asked which made more progress. Their Qwen 3.6 35B verifier used 20 score levels and eight repeated evaluations. It reached 87.4% preference accuracy on 500 sampled pairs, compared with 70.8% for a discrete judge using the same vision-language model. The original RoboReward benchmark primarily reports mean absolute error on five-level human reward labels, so 87.4% should not be treated as a replacement score on the full original leaderboard.^[1][13]

Progress signals and reinforcement learning

The authors also evaluated whether scores rose as an agent moved through a trajectory. On 500 Terminal-Bench trajectories, they reported a Spearman Value-Order Correlation of 0.848 for successful trajectories and 0.769 for failed trajectories. This metric measures monotonic association between step order and score. It is not a calibrated probability of completion, and it does not establish that a threshold can prospectively detect failure at a known error rate.^[1]

In a robotics experiment, verifier-derived rewards were added while fine-tuning a pi-zero policy with DSRL-SAC on one LIBERO ketchup task. Across five seeds, the paper reports reaching matched success targets with about 1.8 times fewer environment steps and a final success rate of 0.76 rather than 0.69. In a separate MATH experiment, adding a verifier reasoning reward while training Qwen3-8B with GRPO reduced the steps to matched accuracy by about 10%, reported as roughly 1.1 times sample efficiency across three seeds.^[1]

These experiments show how the score can be reused as a process reward model or outcome reward, but their scope is narrow. The paper's appendix describes the reinforcement-learning work as single-turn, and the detailed robotics setup covers one task. The results do not establish the same sample-efficiency gains across other robots, environments, reasoning datasets, or verifier models.^[1][16]

Version 0.2.0 and self-verification

Version 0.2.0 added a DeepSeek V4 Flash backend, token accounting, prefix-cache changes, and a separate Terminal-Bench 2.1 self-verification dataset and scripts. The changelog reports that moving the evaluation criterion to the end of the prompt and warming shared prefixes raised the measured cache-hit rate from 5.2% to 78.4% on that configuration, reducing uncached input tokens by about 3.4 times. This is a cache measurement for verifier input, not a matched end-to-end cost comparison with another agent system.^[2][4]

In the later repository experiment, DeepSeek V4 Flash generated mini-swe-agent trajectories and the same model verified them. The revised repository table reports the following results:^[2][5]

Scroll sideways for more →

The 78.7% and 79.4% values are sample averages over different candidate-pool truncations, so their difference is not evidence that the underlying model changed. The 88.0% value is the success rate after the repository's verifier selected one of five stored trajectories. It is not a DeepSeek V4 Flash base-model score, and the experiment was performed and published by the framework's authors rather than an independent evaluator.^[2]

The figures were revised after the 0.2.0 package upload. The initial package README showed 85.4% for best of three and 88.8% for best of five. A repository commit later on August 14 changed them to 86.5% plus or minus 1.1 and 88.0% plus or minus 0.6. The revised figures are the appropriate ones to cite, but the change also makes the source revision part of the result's necessary metadata.^[3][5]

Benchmark version and harness effects

The July paper used Terminal-Bench 2.0, while the August repository experiment used Terminal-Bench 2.1. The benchmark maintainers changed 28 of 89 tasks in version 2.1 because of external-dependency drift, resource mismatches, or misspecified tasks. In their representative runs, moving from 2.0 to 2.1 changed some model-harness scores by several percentage points and one listed score by 12.1 points. The two LLM-as-a-Verifier Terminal-Bench tables therefore cannot be compared as though they used the same test.^[6][7]

An agent benchmark also measures more than the language model. Anthropic reported a six-point spread between its least- and most-resourced Terminal-Bench 2.0 configurations and found that resource enforcement affected both infrastructure failures and the strategies that could complete a task. This does not invalidate selection within a fixed set of already generated trajectories, but it limits comparisons with external leaderboard runs unless the model, harness, resource limits, benchmark revision, and evaluation policy are aligned.^[8]

SWE-bench Verified was constructed as a 500-task, human-filtered subset of real GitHub issues. In 2026, OpenAI audited 138 tasks that o3 did not solve consistently and reported material test or problem-description issues in 59.4% of that audited subset, along with evidence that frontier models had encountered some benchmark material during training. The audit does not target LLM-as-a-Verifier, but it makes small frontier-level score differences on this benchmark less conclusive than a raw percentage suggests.^[9][10][11]

Relation to earlier verifier methods

LLM-as-a-judge work established the use of language models for pointwise and pairwise evaluation and documented position, verbosity, self-enhancement, and reasoning biases. LLM-as-a-Verifier retains an LLM evaluator but uses the full ordered score-token distribution, repeated calls, and multiple criteria to create its reward. Its ring step explicitly alternates candidate positions to address one known source of judge bias.^[1][14]

Earlier trained verifiers generated many mathematical solutions and learned a model that ranked them, establishing a best-of-N pattern. Process-supervision work instead trained reward models on correctness labels for intermediate reasoning steps. LLM-as-a-Verifier differs in that its core scoring method uses an existing logprob-accessible model without task-specific verifier training, although its dense scores can later be used during policy training.^[1][15][16]

V1 is a closer 2026 comparison because it also uses pairwise self-verification and tournament ranking. V1 includes an uncertainty-guided inference algorithm and a reinforcement-learning method that jointly trains a model as generator and verifier. LLM-as-a-Verifier uses V1 as a ranking baseline but centers on expected score-token rewards, criteria and repetition scaling, and a ring-selected pivot tournament.^[1][17]

Limitations and reproducibility

Direct scoring requires access to the verifier's score-token log probabilities, which several restricted model APIs do not expose. The paper proposes a two-stage workaround in which a closed model supplies reasoning and an accessible model converts that reasoning to a continuous score. That workaround adds another model and does not recover the closed model's own hidden score distribution.^[1]

Increasing N, K, C, or G changes the inference budget. More candidates require more generation, while more criteria and repetitions require more verifier calls. Version 0.2.0 records tokens and can exploit provider caching, but it does not publish a matched end-to-end cost study against every system used for comparison. No stable universal cost multiplier can be inferred from its cache-hit measurement.^[2][4]

The framework also depends on the verifier's knowledge, prompt interpretation, and rubric. Fine score resolution can eliminate exact ties while still producing confident errors, repeated evaluations can preserve shared bias, and criteria decomposition can overlook an unlisted requirement. Oracle Pass@N remains unattainable unless the selector is perfect.^[1][14]

As of August 19, 2026, the available evidence supported describing LLM-as-a-Verifier as a documented research framework with reproducible code and author-reported multi-domain experiments. It did not support treating the repository's later 88.0% selection result as independent replication, a new base-model score, or proof of a general cost advantage.^{[1][2][5][7][8]}

References

1. ^Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, and Azalia Mirhoseini. *LLM-as-a-Verifier: A General-Purpose Verification Framework*. arXiv:2607.05391v2, July 7, 2026. arxiv.org/...2607.05391
2. ^LLM-as-a-Verifier authors. *llm-as-a-verifier repository README*. Revision af748b49, August 19, 2026. github.com/...README.md
3. ^Python Package Index. *llm-verifier 0.2.0*. August 14, 2026. pypi.org/...0.2.0
4. ^LLM-as-a-Verifier authors. *Changelog: version 0.2.0*. Release commit 628d8fc, August 14, 2026. github.com/...CHANGELOG.md
5. ^Jacky Kwok. *Revise LLM-as-a-Verifier metrics for Self-Verification*. GitHub commit 115de305, August 14, 2026. github.com/...e305f23ed89bc42e86e010853c40059f3f7d
6. ^Mike A. Merrill et al. *Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces*. arXiv:2601.11868, 2026. arxiv.org/...2601.11868
7. ^Terminal-Bench Team. *Terminal-Bench 2.1*. May 6, 2026. tbench.ai/...terminal-bench-2-1
8. ^Anthropic. *Quantifying infrastructure noise in agentic coding evals*. February 5, 2026. anthropic.com/...infrastructure-noise
9. ^Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* ICLR 2024. arxiv.org/...2310.06770
10. ^OpenAI. *Introducing SWE-bench Verified*. August 13, 2024. openai.com/...introducing-swe-bench-verified
11. ^OpenAI. *Why SWE-bench Verified no longer measures frontier coding capabilities*. March 2026. openai.com/...o-longer-evaluate-swe-bench-verified
12. ^Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, James Zou, Andrew Y. Ng, and Jonathan H. Chen. *MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents*. arXiv:2501.14654v2, 2025. arxiv.org/...2501.14654
13. ^Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn. *RoboReward: General-Purpose Vision-Language Reward Models for Robotics*. arXiv:2601.00675, 2026. arxiv.org/...2601.00675
14. ^Lianmin Zheng et al. *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*. NeurIPS 2023. arxiv.org/...2306.05685
15. ^Karl Cobbe et al. *Training Verifiers to Solve Math Word Problems*. arXiv:2110.14168, 2021. arxiv.org/...2110.14168
16. ^Hunter Lightman et al. *Let's Verify Step by Step*. ICLR 2024. arxiv.org/...2305.20050
17. ^Harman Singh et al. *V1: Unifying Generation and Self-Verification for Parallel Reasoners*. arXiv:2603.04304, 2026. arxiv.org/...2603.04304