Deep Research Bench

Deep Research Bench (DRB) is a benchmark introduced by FutureSearch in May 2025 to evaluate how well large language model agents complete complex research tasks on the open web. Its original paper defines 89 instances across eight task types, pairs them with answers worked out by human researchers, and supplies RetroSearch, an environment that serves previously collected web pages to agents.^[1] The project is a living benchmark: a FutureSearch overview updated in June 2026 reported 91 tasks and an actively maintained leaderboard.^[2]

This article covers the FutureSearch benchmark titled "Deep Research Bench: Evaluating AI Web Research Agents." It is not the similarly named DeepResearch Bench, a separate USTC and Metastone project for grading long-form bilingual research reports with the RACE and FACT frameworks.^[4][5] The two projects have different authors, task sets, evaluation methods, and canonical pages.

DRB tests task-specific research outputs rather than the general quality of a long report. Depending on the task, an answer may be a number, source URL, probability, evidence set, reference class, or structured dataset. The benchmark uses different scoring rules for those outputs, so its paper warns that scores from different task types should not be interpreted as direct measures of relative task difficulty.^[1]

Development and purpose

Deep Research Bench developed from a 2024 FutureSearch study of open-web research agents. That earlier study examined analyst-style tasks inspired by finance, consulting, and forecasting work, and compared several agent architectures and underlying models.^[3] The 2025 benchmark expanded the task suite, added the RetroSearch environment, and evaluated both API-driven agents and commercial research products.^[1]

The paper identifies eight authors, listed alphabetically: Nikos I. Bosse, Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, and Jack Wildman. All were affiliated with FutureSearch. The paper was submitted to arXiv on May 6, 2025.^[1]

Task provenance was mixed. At publication, about 40 percent of instances came from FutureSearch client work, about 25 percent were based on substantial research already available online while the answer-bearing source was blocked from the tested agent, and about 35 percent were created by the benchmark authors. The authors argued that this mixture connected the evaluation to practical research work while still allowing them to vary task type and difficulty.^[1]

The complete task set was not released with the paper. The authors published eight sample instances in an appendix but withheld the rest to reduce the risk of benchmark contamination. Consequently, the public paper documents the design and examples, while access to benchmark runs is mediated through FutureSearch's evaluation service.^[1][6]

Original task suite

The May 2025 paper reported the following distribution. "Instances" refers to the original 89-instance release, not later additions.^[1]

Scroll sideways for more →

For set-valued tasks, the benchmark compares returned items with a curated ground-truth list. Find Dataset and Gather Evidence use recall, generally with an instance-specific cap on the number of returned items. Populate Reference Class and Compile Dataset can use recall alone or combine precision and recall as an F1 score. Validate Claim asks the agent for a probability and scores its normalized absolute distance from a human researcher's probability. Some evaluations use an LLM to recognize equivalent names or judge whether a cited source is at least as reliable as the reference source.^[1]

The eight task types are heterogeneous by design. The paper first averages attempts for each approach and task, but states that a higher score on one task type than another does not show that the first task type is easier. Category breakdowns therefore provide information about a system's profile, while the aggregate score is the main summary used for model comparisons.^[1]

RetroSearch environment

RetroSearch is intended to reduce changes caused by pages being edited, removed, or becoming inaccessible. The benchmark authors built large task-specific collections of web pages and stored both raw HTML and processed text. In the 2025 paper, the median collection size by task type ranged from 9,993 pages for Validate Claim to 110,946 for Gather Evidence. Individual task collections ranged from roughly 4,000 to 189,000 pages.^[1]

A RetroSearch query does not simply search a static local index. According to the paper, it first sends the agent's query to the live Serper search API. It then looks for the returned URLs in the RetroSearch database and other archives, removes results for which no stored page is available, generates replacement snippets from stored content, and returns a response in the Serper format. Page-reading requests return processed content from the collected pages. This freezes the content that an agent reads while trying to preserve an experience similar to live search.^[1]

For its initial experiments, FutureSearch used a ReAct agent with search and document-query tools. Runs were bounded at 50 actions per instance. Non-reasoning models used an observation, thought, and action loop; for reasoning models, the explicit thought step was omitted and action selection relied on the model's internal reasoning. The authors describe this as an elicitation choice for the experiment, not a requirement imposed on all systems evaluated with DRB.^[1]

Commercial research products could not be connected to RetroSearch, so they were tested on the live web. They also could not be evaluated on the roughly 25 percent of instances that depended on blocking an answer-bearing page. The paper's combined product comparison therefore used 66 instances, whereas the principal API-agent results used all 89.^[1]

Initial evaluation

The initial study tested agents based on models available by May 2025 and a selection of commercial web-research products. Each custom agent was run twice per instance; each commercial product was run once. Model calls used temperature zero when the provider supported it, and results were collected over about two months. These choices reduce some variability but do not make the runs deterministic.^[1]

On the live-web custom-agent evaluation, the highest reported aggregate score was 0.51 for an o3-based agent. The authors gave a rough estimate of about 0.8 for the benchmark's noise ceiling, reflecting subjective judgments, possible answer errors, and LLM-assisted scoring. That 0.8 value was an author estimate, not a separately measured human score.^[1]

Live and RetroSearch results were broadly similar in ordering but not identical. In the paper's comparison, o3 scored 0.51 live and 0.46 with RetroSearch, while Gemini 2.5 Pro scored 0.45 live and 0.46 with RetroSearch. The authors attributed differences to run-to-run variation, gaps in the stored corpus, and changes between the frozen collection and the live web. They treated future RetroSearch results as the more stable basis for longitudinal comparison.^[1]

In the product comparison, ChatGPT with o3 and web search scored above the tested OpenAI Deep Research product, and standard Perplexity Pro scored above Perplexity Deep Research. The pattern did not generalize to every provider: Gemini Deep Research performed above Gemini 2.5 Pro with search, and Claude Research with extended thinking performed slightly above Claude 3.7 with web search. The paper explicitly cautions that each product was run only once per instance and that the sample was small.^[1]

Trace analysis

DRB also evaluated selected agent traces for tool-call hallucination, repeated tool use, and forgetting previously observed information. This analysis covered six model configurations and only 16 to 32 traces per configuration because the automated checks were expensive. The authors excluded other proposed failure labels that they could not classify with sufficient reliability.^[1]

In a linear regression using the three retained failure rates, forgetting had the largest negative coefficient, at -0.843 with a reported p-value of 0.014. Hallucination had a coefficient of -0.653 with p = 0.055, and repeated tool calls had a coefficient of -0.165 with p = 0.626. The regression explained little of the score variation: the reported R-squared values were 0.093 on the training set and 0.129 on the test set. The -0.843 figure is therefore a regression coefficient in this exploratory model, not a correlation coefficient between forgetting and final score.^[1]

Manual trace review found recurring problems with premature stopping, weak search-query formulation, failure to revisit an initial strategy, and accepting superficially relevant evidence. The authors bound these observations to their simple, low-elicitation prompts and did not claim that the same rankings would necessarily hold under more intensive prompting or different agent scaffolding.^[1]

Limitations and interpretation

Several constraints affect how DRB results should be read:

• The original release had 89 instances, and some task types contained fewer than ten. The paper provides aggregate evidence but limited statistical resolution for narrow category comparisons.^[1]
• Ground truth is not uniformly objective. Validate Claim uses human probability assessments, some list tasks depend on judgments about prominence or completeness, and some scoring decisions use an LLM. The authors note that a better-supported answer can in principle disagree with the stored human answer.^[1]
• The full instance set and stored corpora were not openly released with the paper. Withholding them may reduce contamination, but it also limits independent replication and inspection.^[1]
• RetroSearch freezes retrieved page content, but its documented search procedure begins with a live search result set. It also cannot guarantee that its crawl includes every useful source or every misleading distractor that a live-web agent might encounter.^[1]
• The controlled API removes some failures found on the live web, such as intermittent outages and bot blocking. Future models may also contain more of the snapshot's facts in their training data, which the authors identify as a reason to create new snapshots periodically.^[1]
• The commercial products used the live web, were evaluated on only 66 eligible instances, and were run once. Their outputs sometimes required manual correction or a second formatting prompt before scoring.^[1]

The benchmark has continued to change since the paper. FutureSearch's June 2026 overview reported 91 tasks and says that models and tasks are added over time.^[2] Results from the 89-instance May 2025 study should therefore be labeled as initial-paper results rather than assumed to describe the current leaderboard.

References

1. ^Bosse, Nikos I., Jon Evans, Robert G. Gambee, Daniel Hnyk, Peter Mühlbacher, Lawrence Phillips, Dan Schwarz, and Jack Wildman. "Deep Research Bench: Evaluating AI Web Research Agents." arXiv:2506.06287, submitted May 6, 2025. arxiv.org/...2506.06287
2. ^FutureSearch. "Deep Research Bench Leaderboard: LLM Web Research Agent Rankings." Published June 25, 2025; updated June 25, 2026. futuresearch.ai/deep-research-bench
3. ^Mühlbacher, Peter, Nikos I. Bosse, and Lawrence Phillips. "Towards a Realistic Long-Term Benchmark for Open-Web Research Agents." arXiv:2409.14913, revised September 25, 2024. arxiv.org/...2409.14913
4. ^Du, Mingxuan, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. "DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents." arXiv:2506.11763, submitted June 13, 2025. arxiv.org/...2506.11763
5. ^DeepResearch Bench project. "DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents." deepresearch-bench.github.io
6. ^FutureSearch. "FutureSearch Evals." drb.futuresearch.ai