BigFinanceBench

Scroll sideways for more →

BigFinanceBench (BFB) is a benchmark for evaluating how well AI agents perform the research work of professional financial analysts, developed by the finance-AI company Rogo and announced on May 27, 2026.^[1] It consists of 928 open-ended, expert-authored questions spanning capital structure and returns, M&A, valuation, private capital, metrics computation, earnings analysis, and scenario forecasting, and it grades each answer against a point-weighted rubric that scores the full derivation rather than only the final number.^[1][2] Rogo describes the benchmark as "workflow-grounded, agent-executed, derivation-graded," positioning it as a different category from earlier finance benchmarks such as FinanceBench, which grades short-answer extraction against fixed passages, or FinQA, FinanceMath, and DocFinQA, which grade arithmetic against single contexts.^[1][12]

The benchmark drew attention beyond Rogo's own customer base within weeks of release: OpenAI cited it on the launch page for GPT-5.6 in July 2026, and Rogo maintains a public leaderboard at bigfinancebench.com that it updates as new models ship.^[3][6]

Background

Rogo is a New York based company that sells an agentic AI-in-finance platform to investment banks, private equity firms, and asset managers; it raised a $160 million Series D at a $2 billion valuation in April 2026.^[4] The company says it built BigFinanceBench because customers kept asking which model to use, and existing AI benchmarks could not answer that question for finance: correctness in analyst work depends on layers of context (which source, which period, which accounting definition, which assumptions), and prior finance benchmarks did not measure the derivation at all.^[1]

The accompanying paper, "BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents," was posted to arXiv on June 2, 2026. Its authors are listed under Rogo and OpenAI affiliations, with a note that the OpenAI-affiliated authors did the work while at Rogo.^[2]

Design

The 928 questions were written by 52 financial-research practitioners, predominantly former investment bankers and private equity investors, and every question was stress-tested by a panel of 12 senior reviewers.^[1][2] All questions are designed to have an objective answer, to be time-constrained so the answer stays fixed over time (for example "2Q25" rather than "last quarter"), and to require multi-step reasoning that often draws on multiple sources.^[1] The questions divide into four workflow groups:^[1]

Scroll sideways for more →

Every question carries a grading rubric written by the same practitioners, averaging 17 line items tagged as Retrieval, Definition, or Calculation and weighted from 1 to 10 points. Across the benchmark there are 15,656 rubric criteria worth 36,241 weighted points.^[1][2] A model earns partial credit for finding the right filing even if it later miscalculates, and loses credit at the specific step where it failed, which lets the benchmark localize failures across the analyst workflow.^[1][2]

Evaluation harness and scoring

Models run each question three times in a minimal open-source agent harness with access to SEC EDGAR, the public web, and a Python sandbox, so that scores measure the model rather than the scaffolding.^[1][3] Traces are graded by two LLM judges, Gemini 3.1 Pro and Claude Opus 4.7, with scores averaged across judge model families to limit self-preference.^[3] The harness used in the paper is published on GitHub under an Apache 2.0 license.^[5]

Rogo reports two headline metrics: final-answer accuracy (whether the response matches the expert answer) and rubric score (weighted credit across the derivation). In the launch evaluation of ten frontier and open-weight systems, rubric scores exceeded final-answer accuracy by roughly 16 percentage points on average, which Rogo argues shows how much reasoning is invisible to binary grading: models frequently retrieve the correct filings and follow the right logic but lose everything to a final-step arithmetic slip.^[1][2]

Findings

The paper's launch results showed substantial headroom: the best single system (Claude Opus 4.7 or GPT-5.5, which were nearly tied with Claude Sonnet 4.6) reached only 58.8 percent rubric score.^[1][2] No model led across the dataset; each of the top systems excelled in a different class of financial work, with GPT-5.5 strongest on capital structure and M&A, Sonnet 4.6 on earnings quality and statement analysis, and Opus 4.7 on private capital and forecasting.^[1] Rogo also evaluated routing strategies: a coarse router selecting models by workflow and source type gained 4.5 rubric points over the best single model, and a best-of-ten oracle gained 13.2 points, which the company reads as evidence that orchestration matters as much as raw model capability in financial deployments.^[1][2]

Leaderboard

The public leaderboard at bigfinancebench.com is updated as new models ship. As of its August 3, 2026 update it covered 28 models, led by Meta's Muse Spark 1.1.^[3] The top ten by rubric score:

Scroll sideways for more →

Because every listed model runs the same minimal harness, the leaderboard does not include commercial agent products built on top of those models.^[3]

Adoption and the Primer episode

OpenAI's GPT-5.6 launch page featured BigFinanceBench in July 2026: in a testimonial quote on the page, Rogo's Alex Wang reported that GPT-5.6 improved rubric quality by 6.2 points and answer accuracy by 3.6 points over GPT-5.5 on the benchmark.^[6]

In a July 29, 2026 blog post, Primer, an AI agent for equity research whose team is led by founder and portfolio manager Seb Jory, reported running its product against BigFinanceBench's 50-question public subset using Rogo's own three-trial, two-judge protocol.^[7][8] Primer, which at the time ran on GPT-5.5 as its underlying model, reported 79.1 percent final-answer accuracy against 55.8 percent for GPT-5.5 alone in the benchmark's minimal harness, scored on 43 of the 50 public questions after excluding seven whose reference answers Primer disputed; the company attributed the roughly 23-point gap entirely to its harness, since the underlying model was the same.^[7] The claim, publicized by Primer's head of applied AI Alistair Smallwood in August 2026, is self-reported and covers only the public subset: Primer does not appear on Rogo's official leaderboard, which benchmarks bare models rather than commercial products.^[7][9][3]

Primer's post also documented what it said were errors in the public subset, including a Netflix cost-per-subscriber reference answer off by three orders of magnitude due to a units slip, a private equity waterfall reference that paid managers more than the stated carried interest could allow, and one question that none of 136 recorded judge verdicts had ever passed.^[7] No public response from Rogo to the error claims had been identified as of August 7, 2026, though the Hugging Face dataset received updates on August 6, 2026 adding human workpapers and GPT-5.5 grades.^[10]

Public release

A 50-question stratified subset of the benchmark is published on Hugging Face under a CC BY 4.0 license, with full reference answers, point-weighted rubrics, traces, and grades; the dataset carries a canary string and do-not-train tags to limit benchmark contamination.^[10] Rogo says the subset is calibrated against the full 928-question set so that rankings on it closely track the full benchmark.^[1] The full question set remains private. Other finance-focused evaluations in the same space include Vals AI's Finance Agent v2, a proprietary 927-question benchmark of entry-level analyst tasks.^[11]

See also

• Rogo
• Vals AI
• FinanceBench
• AI in finance
• Model evaluation
• AI agents
• Benchmark contamination

References

1. ^Rogo, "Rogo's Big Finance Bench," rogo.ai, May 27, 2026. rogo.ai/...introducing-the-big-finance-benchmark
2. ^Alex Wang, Georg Meinhardt, Jacob Katz, Joseph H. Kim, Pratyush K. Chaudhary, Chase Blagden, and Eric Xu, "BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents," arXiv:2606.03829, June 2, 2026. arxiv.org/...2606.03829
3. ^"BigFinanceBench," leaderboard website (August 3, 2026 update). bigfinancebench.com
4. ^Rogo, "Rogo Raises $160M Series D to Scale the Agentic Platform for Finance," PR Newswire, April 29, 2026. prnewswire.com/...c-platform-for-finance-302756546
5. ^Rogo Technologies, "big-finance-benchmark" (reference harness), GitHub. github.com/...big-finance-benchmark
6. ^OpenAI, "GPT-5.6," openai.com, July 2026. openai.com/...gpt-5-6
7. ^Alistair Smallwood, "Primer tops BigFinanceBench," Primer blog, July 29, 2026. primerapp.com/...primer-tops-bigfinancebench
8. ^Primer, "About," primerapp.com. primerapp.com/about
9. ^Al Smallwood (@SmallwoodAl_), post on X, August 6, 2026. x.com/...2085279279533719710
10. ^"RogoAI/big-finance-benchmark," dataset repository, Hugging Face. huggingface.co/...big-finance-benchmark
11. ^Vals AI, "Finance Agent v2." vals.ai/...fabv2
12. ^Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen, "FinanceBench: A New Benchmark for Financial Question Answering," arXiv:2311.11944, November 2023. arxiv.org/...2311.11944