DeepResearch Bench

DeepResearch Bench is a benchmark for evaluating long-form reports produced by web research agents. It was introduced by researchers affiliated with the University of Science and Technology of China and Metastone Technology in a paper submitted to arXiv in June 2025. This article covers that 100-task USTC/Metastone project. It is distinct from FutureSearch's similarly named Deep Research Bench, which has 89 tasks and uses a frozen web environment called RetroSearch.^[1][6]

The benchmark tests systems that plan repeated searches, retrieve information, and synthesize cited reports rather than return a short answer. Its 100 prompts cover 22 fields and are evenly divided between Chinese and English. DeepResearch Bench pairs the task set with two evaluation methods: RACE grades report quality relative to a reference report, while FACT measures how many cited claims are supported by the pages attached to them. The project publishes its task data and evaluator code, but its scores depend on the judge models and on web pages that can change after a report is generated.^[1][3]

The original paper and the current repository provide the following project details.^[1][3][4]

Scroll sideways for more →

Scope and naming

The spelling of the name matters. DeepResearch Bench, with "DeepResearch" closed up, is the USTC/Metastone project described in arXiv:2506.11763. The separate FutureSearch paper uses "Deep Research Bench" with a space and is arXiv:2506.06287. FutureSearch evaluates agents against worked-out answers for 89 multi-step research instances in eight categories. RetroSearch gives those agents a previously scraped collection of web pages so that later runs can search the same material.^[6]

The USTC/Metastone benchmark instead evaluates the reports themselves. Its task prompts ask for open-ended research and synthesis. RACE compares a generated report with a reference across task-specific quality criteria. FACT follows the report's cited URLs and checks whether the retrieved text supports the statements attached to those URLs. The two projects therefore differ in their datasets, evaluation targets, and treatment of the web. They should not share results, descriptions, or leaderboard claims.

DeepResearch Bench II is also a separate dataset, even though it came from an overlapping research group. It uses expert-written source reports to derive fine-grained binary rubrics. The original project's repository says that DeepResearch Bench continues to be maintained because the two benchmarks have different evaluation aims.^[3][5]

Benchmark construction

Demand sample and topic distribution

The authors started with 96,147 raw queries from interactions with a web-search-enabled large language model chatbot. According to the paper, the logs were anonymized by removing information such as user IDs, IP addresses, and session metadata. The authors defined a deep research task as one that requires multiple rounds of web search, information gathering, analysis, and production of a report.^[1]

They used DeepSeek V3, specifically the DeepSeek-V3-0324 version, to filter the raw collection. The model classified 44,019 queries, about 45.8 percent of the starting pool, as matching the project's definition. It then assigned those filtered queries to 22 domains selected from the WebOrganizer taxonomy. The resulting counts supplied a topic distribution intended to approximate demand in that chatbot's query pool.^[1]

This procedure was not a direct conversion of user logs into benchmark prompts. Running and evaluating web research systems was costly, so the authors compressed the observed domain proportions into a set of 100 tasks. Half were written in Chinese and half in English. The paper presents the task set as preserving the broad topical balance of the filtered query collection, subject to rounding and the small final sample.^[1]

The 22 domains include science and technology subjects, finance and business, software use and internet topics, art and design, entertainment, history, industry, transportation, and travel. The paper's domain chart gives science and technology the largest combined share. It groups low-frequency fields into an "Others" segment in one visualization but retains the 22 individual labels in the accompanying distribution.^[1]

Expert task collection

The authors invited PhD holders or senior practitioners with more than five years of relevant experience to propose candidate tasks. The project team manually screened submissions for quality, clarity, complexity, and consistency with its definition of deep research. Appendix A says that each accepted task was developed by a verified PhD-level expert who received orientation on the task specifications and participated in multiple rounds of refinement.^[1]

The accepted prompts are research assignments, not conventional fact questions. Examples in the paper request comparative analysis, evidence from multiple types of sources, calculations, and a structured report. A system must decide how to search, choose material, reconcile sources, and present the result. That format separates the benchmark from short-form question answering, even though both may require retrieval.

The source paper does not establish that 100 different people wrote the 100 tasks, and it does not describe the Chinese and English tasks as paired translations. The verified description is narrower: the set contains 100 expert-developed tasks, divided evenly by language, with topic counts derived from the filtered query distribution.^[1]

RACE report evaluation

RACE stands for Reference-based Adaptive Criteria-driven Evaluation framework with Dynamic Weighting. It is an LLM-as-a-judge method designed for open-ended reports whose requirements vary from task to task. Four top-level dimensions remain fixed:^[1]

1. Comprehensiveness measures coverage of the relevant parts of the requested topic.
2. Insight/Depth measures analysis of causes, effects, relationships, and other material beyond surface description.
3. Instruction-Following/Relevance measures whether the report answers the assigned question and observes its constraints.
4. Readability measures structure, clarity, and presentation.

For a given task, the judge first produces weights for those dimensions over repeated trials. It then generates more specific criteria within each dimension and assigns criterion weights. The judge scores both the target report and a high-quality reference report against the same criteria. Those criterion scores become weighted dimension scores and then an intermediate score for each report.^[1]

The final target score is relative:

S_final(R_target) =
  S_int(R_target) /
  (S_int(R_target) + S_int(R_reference))

This formula changes how a RACE number should be read. A score near 50 means that the target and reference received similar intermediate scores under the generated criteria. It does not mean that half of a fixed set of answers was correct. The original paper advises interpreting rank and proportional differences rather than treating the values as ordinary percentage grades.^[1]

The 2025 experiments used Gemini 2.5 Pro Preview as the RACE judge. The reference reports were produced by the Gemini 2.5 Pro Deep Research product available in April 2025. That choice supplies a consistent comparison point, but it also makes the result dependent on a model-generated reference and model-generated criteria.^[1]

Human-consistency study

The authors compared RACE with expert judgments on all 50 Chinese tasks. Four agent reports were available for each task, and three domain-expert annotators evaluated the four reports. The study involved more than 70 annotators who held master's degrees and had relevant domain expertise. Each person handled no more than three queries. The paper estimates about 1.5 hours per query for evaluating its four reports and about 225 person-hours in total.^[1]

The study used four consistency measures. Pairwise Agreement Rate counted how often the evaluator and humans preferred the same report in a pair. Overall Pearson Correlation compared average scores for the four tested agents. Two filtered task-level correlations were calculated after removing tasks with negative intraclass correlation coefficients, leaving 37 of the 50 tasks.^[1]

Scroll sideways for more →

The values are the paper's reported 0-100-scaled consistency results. RACE Full had higher pairwise agreement than the direct-scoring prompt and the version without a reference. Its 99.54 Overall Pearson figure describes correlation among four agent-level averages; it is not a 99.54 percent item-level accuracy rate. The smaller filtered correlations show why the paper reports several measures rather than relying on that aggregate alone.^[1]

FACT citation evaluation

FACT stands for Framework for Factual Abundance and Citation Trustworthiness. It examines the relationship between claims and citations in a generated report. The original pipeline has four main stages:^[1]

1. A judge extracts discrete statement and cited-URL pairs.
2. It removes repeated pairs when multiple statements attached to the same URL express the same fact.
3. The Jina Reader API retrieves text from each cited web page.
4. A judge labels each pair "support" or "not support" according to whether the retrieved page text provides evidence for the statement.

FACT reports two quantities. Citation Accuracy is calculated for each task as the fraction of its unique statement-URL pairs judged supported, then averaged across tasks. A report with no cited pairs receives zero for that task. Average Effective Citations per Task is the number of supported unique pairs across the evaluation divided by the number of tasks. A system can therefore retrieve many supported statements while still having a lower support rate, or cite fewer statements with a higher support rate.^[1]

The original implementation used Gemini 2.5 Flash for pair extraction, deduplication, and support judgment. The authors manually checked a random sample of 100 statement-URL pairs. They reported that the model agreed with human "support" decisions in 96 percent of cases and with human "not support" decisions in 92 percent of cases.^[1]

The name FACT can be misleading if it is read as a general truth score. The evaluator asks whether the fetched page supports the attached statement. It does not independently prove that the page or statement is factually correct. DeepResearch Bench II makes this distinction explicit, noting that an unofficial page may support a false claim.^[5] FACT is therefore a citation-support measure, not a substitute for source-quality review or independent fact checking.

Results in the original paper

The original paper evaluated four commercial research products and twelve general models supplied with search tools. Because commercial systems can change without a public version identifier, the authors recorded when each set of reports was generated. OpenAI Deep Research outputs were collected from 1 April through 8 May 2025. Gemini 2.5 Pro Deep Research, Perplexity Deep Research, and Grok Deeper Search had collection windows in April 2025. Outputs from general search-enabled models were collected from 11 through 13 May 2025, depending on provider.^[1]

The following values are historical results from those output windows under the paper's original Gemini-based evaluator. They are not a current leaderboard and should not be compared directly with results produced by the repository's later GPT-based evaluator.^[1][3]

Scroll sideways for more →

Gemini's product received the highest overall RACE value among those four 2025 product outputs, while Perplexity's had the highest Citation Accuracy in that subset. Gemini also had far more Effective Citations per Task. The separation between support rate and supported-citation count is one reason the benchmark publishes the FACT quantities separately rather than combining them into a single number.^[1]

The paper's search-tool group shows the same distinction. Claude 3.5 Sonnet with Search had 94.04 percent Citation Accuracy but 9.78 Effective Citations per Task, while Claude 3.7 Sonnet with Search had 93.68 percent Citation Accuracy and 32.48 Effective Citations. These are results for the tested configurations and dated output collection, not permanent properties of the model names.^[1]

Software and current maintenance

The official repository contains the prompt data, task-specific criteria, raw and cleaned report directories, evaluator prompts, utilities, and runnable RACE and FACT scripts. Its README requires Python 3.9 or later. A model output file uses JSON Lines, with id, prompt, and article fields for each task. The evaluation client can call a judge through OpenRouter or OpenAI, and FACT additionally requires a Jina API key to retrieve cited pages.^[3]

The repository's LICENSE file contains the Apache License 2.0. This license covers the released repository material under its terms; it does not imply that third-party web pages fetched during an evaluation share that license.^[4]

The project also maintains an official site and links a public leaderboard hosted on Hugging Face.^[2][3] Leaderboard entries can change as systems, submissions, and evaluators change. Reproducing a result therefore requires the generated reports, the evaluator generation, and the relevant configuration, not only the displayed model name.

Evaluator change in 2026

The original paper and its 2025 scores used Gemini 2.5 Pro for RACE and Gemini 2.5 Flash for FACT. In a repository update dated 11 May 2026, the maintainers changed the default judges to GPT-5.5 for RACE and GPT-5.4-mini for FACT after testing replacement candidates on the human-annotated subset. The previous evaluator code remains available on a Gemini-2.5 branch.^[3]

The same update announced separate leaderboards during the transition so that entries would be compared only with results produced by the same evaluator generation. It also introduced a second version of the evaluation pipeline with chunk-based article cleaning for very long reports. These changes mean that a bare "DeepResearch Bench score" is incomplete. A reported result should identify at least the benchmark version or task set, evaluator generation, report-generation date, and whether it came from the legacy or current pipeline.^[3]

Follow-up work and research use

DeepResearch Bench II was posted in January 2026 and released by the project in February. It contains 132 tasks across 22 domains and 9,430 binary rubrics derived from expert-written reports. The authors report more than 400 hours of expert work. Its three evaluation dimensions are information recall, analysis, and presentation. This rubric-based design diagnoses which expected pieces of information and analysis appear in a report, while the original RACE method compares a target with a reference under generated criteria.^[3][5]

The follow-up paper also documents lessons from the first benchmark. It identifies possible anchoring bias when an LLM generates criteria from a reference and then participates in scoring. It also states that citation support does not prove the underlying information true. These observations refine how the original RACE and FACT metrics should be interpreted rather than invalidating the original task set.^[5]

Other research groups have used DeepResearch Bench as an evaluation set. FS-Researcher, published in the ACL 2026 long-paper proceedings, reports experiments on DeepResearch Bench and DeepConsult for a file-system-based research-agent architecture.^[7] The ACL 2026 industry paper "Don't Stop Early" also evaluates a controlled information-flow and evidence-aware termination method on the public benchmark.^[8] These papers show use outside the benchmark's author group without establishing that it is a universal standard for agent evaluation.

Limitations

The final set has only 100 tasks. The authors attribute this size to the effort needed for expert task development, repeated refinement, running research systems, and evaluating long reports. Compressing 44,019 filtered query classifications into 100 prompts preserves only a coarse approximation of the source pool's domain proportions. It cannot sample every kind of research request within each of 22 fields.^[1]

The demand distribution comes from one in-house chatbot query collection and depends on DeepSeek-V3-0324 for filtering and classification. The paper describes anonymization and the selected taxonomy, but the benchmark's topical mix still reflects that source population, its time period, the project's deep-research definition, and the classifier's decisions. The 50 Chinese and 50 English tasks test two languages; results do not establish performance in other languages.^[1]

RACE depends on generated criteria, task-specific weights, a judge model, and a model-produced reference report. The human-consistency experiment covered the 50 Chinese tasks and four agent outputs, with three annotators per task. It did not validate every possible judge, report style, agent, or English task. The same group's follow-up paper describes potential anchoring from model-generated criteria and references.^[1][5]

The original RACE reference reports came from Gemini 2.5 Pro Deep Research, and the judge was Gemini 2.5 Pro Preview. A later change of judge can alter scores even when the target reports stay fixed. Historical and current leaderboard values therefore require evaluator labels and should not be placed in one ranking without re-evaluation under a common setup.^[1][3]

FACT relies on retrieving the current text at a cited URL. Pages may change, disappear, block automated retrieval, or present different content to the reader. A support decision also says nothing about whether a source is authoritative. A well-cited false page can support a false statement, while a correct statement can lose credit if its page is unavailable or the relevant text is not retrieved. FACT should be interpreted alongside manual source review, not as a complete measure of hallucination or factual correctness.^[1][5]

Finally, the original product results describe reports collected in April and May 2025. Commercial systems with the same product names may have changed since those runs. The repository's 2026 evaluator and cleaning changes add another source of variation. Reproducible comparisons need preserved reports and a named evaluation configuration; a model name and a single score are insufficient.^[1][3]

References

1. ^Du, Mingxuan, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. "DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents." arXiv:2506.11763, submitted 13 June 2025. arxiv.org/...2506.11763
2. ^DeepResearch Bench project. "DeepResearch Bench." Official project website. deepresearch-bench.github.io
3. ^Ayanami0730. "deep_research_bench." Official GitHub repository and README, accessed 28 July 2026. github.com/...deep_research_bench
4. ^Ayanami0730. "Apache License, Version 2.0." Repository license file, accessed 28 July 2026. raw.githubusercontent.com/...LICENSE
5. ^Du, Mingxuan, et al. "DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Report." arXiv:2601.08536v2, 30 January 2026. arxiv.org/...2601.08536
6. ^FutureSearch. "Deep Research Bench: Evaluating AI Web Research Agents." arXiv:2506.06287, 2025. arxiv.org/...2506.06287
7. ^Zhu, Chiwei, et al. "FS-Researcher: Test-Time Scaling for Long-Horizon Research Tasks with File-System-Based Agents." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, 2026. aclanthology.org/2026.acl-long.288
8. ^Choubey, Prafulla Kumar, et al. "Don't Stop Early: Scalable Enterprise Deep Research with Controlled Information Flow and Evidence-Aware Termination." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics: Industry Track, 2026. aclanthology.org/2026.acl-industry.116