# Online-Mind2Web

> Source: https://aiwiki.ai/wiki/online_mind2web
> Updated: 2026-07-28
> Fact-checked: 2026-07-28
> Categories: AI Agents, AI Benchmarks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Online-Mind2Web." aiwiki.ai, 28 Jul 2026. https://aiwiki.ai/wiki/online_mind2web
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

Online-Mind2Web is a live-web [benchmark](https://aiwiki.ai/wiki/benchmark) for measuring whether browser-based [AI agents](https://aiwiki.ai/wiki/ai_agents) can complete multi-step tasks on real websites. Tianci Xue and collaborators introduced it in the paper "An Illusion of Progress? Assessing the Current State of Web Agents," first posted in April 2025 and published at the 2025 Conference on Language Modeling (COLM). The benchmark began with 300 tasks across 136 websites. Unlike a frozen test set, however, it is maintained by replacing tasks that become invalid, so a result is meaningful only together with its evaluation date, task revision, agent configuration, and scoring method.[1][2][3]

The benchmark was built from the lineage of [Mind2Web](https://aiwiki.ai/wiki/mind2web), but it is not simply the original dataset replayed online. Its creators rechecked and edited older tasks, incorporated a subset from Mind2Web-Live, and wrote new tasks for high-traffic sites. They also introduced WebJudge, a screenshot-and-action-history evaluator intended to make live evaluation less dependent on manual review. The paper reports human results for six agent systems and evaluates several WebJudge configurations against those human labels.[1]

Online-Mind2Web therefore measures a particular operational capability: completing specified browser tasks from a designated starting website under a stated protocol. It does not, by itself, measure all computer use, autonomous research, safety, or reliability across repeated attempts. Its live setting improves ecological realism, while making strict reproduction and score comparison harder than on a fixed offline or self-hosted environment.[1][3]

## Scope and relationship to Mind2Web

The original Mind2Web dataset contains 2,350 tasks across 137 websites and 31 domains. It records crowdsourced demonstrations against saved webpages and tests cross-task, cross-website, and cross-domain generalization. That design makes the examples reusable for training and offline evaluation, but the saved state can diverge from a website that has since changed.[5]

Online-Mind2Web instead executes instructions against current websites. The paper reports that the authors randomly inspected 650 original Mind2Web tasks and found 47 percent to be invalid or to have outdated reference trajectories. They excluded ambiguous instructions, tasks made impossible by site changes, and sites whose CAPTCHA defenses prevented execution. The resulting 300-task set combined four sources:[1]

| Source | Tasks | Treatment |
|---|---:|---|
| Original Mind2Web | 167 | Selected as distinct, still-executable tasks |
| Rewritten Mind2Web tasks | 24 | Requirements or wording changed to restore solvability or remove ambiguity |
| Mind2Web-Live | 34 | Adapted live tasks incorporated into the new benchmark |
| Newly authored tasks | 75 | Manually created for high-traffic websites |
| Total | 300 | Distributed across 136 websites |

Mind2Web-Live is itself an online evaluation resource associated with the WebCanvas framework. Online-Mind2Web borrows 34 tasks from it but remains a separate benchmark with its own curation, task mix, protocol, evaluator, and leaderboard.[1][6] The name "Online-Mind2Web" should consequently be read as lineage, not as a claim that every original Mind2Web example has a corresponding live task.

## Task construction

Each item supplies a natural-language instruction, a target website, and a human reference length. Tasks cover activities such as filtering products, looking up housing or travel information, and navigating food or transportation services. The paper groups them into broad domains that also include finance, health, education, government services, jobs, entertainment, and technology. Most newly written tasks target popular sites, with site traffic estimated using Similarweb; the appendix also notes a smaller number of niche or non-US sites.[1]

Difficulty is based on the number of actions in a human reference trajectory, rather than on an independent cognitive-complexity rating:[1]

| Difficulty | Human reference length | Tasks |
|---|---:|---:|
| Easy | 5 actions or fewer | 83 |
| Medium | 6 to 10 actions | 143 |
| Hard | 11 actions or more | 74 |
| Total |  | 300 |

This definition is practical but incomplete. A short task can still fail because of authentication, localization, a transient modal, or an unusual site control. A longer reference can sometimes be shortened by a different valid path. Reference length is therefore a stratification rule, not a universal measure of task difficulty.

The authors also tested whether search could shortcut the intended navigation. A simple search agent solved 51 percent of a stratified 100-task sample from WebVoyager, compared with 22 percent of a similarly stratified Online-Mind2Web sample. On the latter sample, success was approximately 50 percent for easy tasks, 18 percent for medium tasks, and 3 percent for hard tasks. This experiment supports the narrower conclusion that the sampled Online-Mind2Web tasks were less amenable to that search shortcut. It does not establish that every task is shortcut-proof or that the two complete benchmarks differ by the same amount.[1]

## Evaluation protocol

In the paper's main experiment, each agent started at the benchmark's specified website and was told not to use Google Search. This constraint was intended to keep agents on the target site and reduce variation caused by switching to alternative sources. Six systems were evaluated: SeeAct, Agent-E, Browser Use, Claude Computer Use 3.5, Claude Computer Use 3.7, and OpenAI Operator.[1]

The comparison is at the system level, not a clean comparison of underlying models. SeeAct, Agent-E, and [Browser Use](https://aiwiki.ai/wiki/browser_use) used GPT-4o-2024-08-06 in the reported setup and were limited to 25 steps. Claude Computer Use 3.7 used Claude-3-7-sonnet-20250219, had a 50-step limit, and ran without its thinking mode. [OpenAI Operator](https://aiwiki.ai/wiki/openai_operator) could be tested only through OpenAI's hosted interface, from which the researchers collected screenshots and action descriptions. The systems also differed in whether they captured full-page or viewport screenshots and in how their actions were represented.[1]

Human judgments were the reference for the paper. At least two annotators independently reviewed every trajectory using the instruction, screenshots, and action history; a third resolved disagreements. The released labels distinguish failure, success, and not executable. The third state covers external or system barriers such as an inaccessible site, a CAPTCHA, a tool failure, or a task invalidated by a site update.[1][3]

These details matter when interpreting the headline success rate. A score can change because the browser agent improved, but also because the model, scaffold, step budget, website state, execution environment, task revision, or judge changed. Online-Mind2Web does not normalize all of those variables into a model-only metric.

## WebJudge

WebJudge is an [LLM-as-a-judge](https://aiwiki.ai/wiki/llm_as_a_judge) pipeline that assigns a binary success or failure label from the task, the factual action history, and trajectory screenshots. The paper deliberately excludes an agent's hidden reasoning and final response from the standard input. Not every system exposes those fields, and a plausible final answer can claim success even when the recorded browser state does not support it.[1]

The pipeline has three stages:[1]

1. It derives key completion points from the task description.
2. It summarizes each screenshot, scores its relevance from 1 to 5, and retains screenshots meeting a threshold. The reported experiments use a threshold of 3.
3. It judges the outcome from the instruction, key points, selected screenshots, and action sequence.

The paper uses a chain-of-thought outcome mode for its primary experiments. A stricter key-point-by-key-point variant performed better on tasks with three or fewer generated points but worse when there were more than three, partly because generated points could be unnecessary or inaccurate. This is one reason the system's agreement rate should not be treated as proof that any individual automatic label is correct.[1]

WebJudge-7B changes the expensive middle stage. It fine-tunes a screenshot relevance model based on [Qwen2.5-VL-7B](https://aiwiki.ai/wiki/qwen2_5_vl), using trajectories from SeeAct, Browser Use, and Claude Computer Use 3.5 for training while holding out Agent-E, Claude Computer Use 3.7, and Operator for evaluation. An external language model still performs key-point generation and the final outcome judgment. WebJudge-7B is therefore not a fully local end-to-end judge; its contribution is reducing screenshot processing to a local model so that the other stages require a fixed two language-model calls per trajectory.[1][10]

The paper reports two related statistics: agreement rate, the percentage of automatic labels matching human labels, and success-rate gap, the difference between aggregate automatic and human success rates. They answer different questions. A judge could have a small aggregate gap while disagreeing on many individual tasks if false positives and false negatives cancel out.

| Evaluator configuration | Mean agreement with human labels | Mean success-rate gap |
|---|---:|---:|
| WebJudge with GPT-4o | 83.6% | 7.8 points |
| WebJudge with o4-mini | 85.7% | 3.8 points |
| WebJudge-7B pipeline | 87.0% | 3.9 points |

The 87.0 percent value is an average across the six agent trajectory sets, not a general accuracy guarantee for arbitrary websites or future agents. In three repeated GPT-4o evaluator runs, the reported standard deviation of agent-level success rates averaged 1.1 percentage points and reached 1.8 points at most.[1]

## Results reported in the paper

The COLM paper's primary table reports human-evaluated success rates for the six tested system configurations:[1]

| Agent configuration | Easy | Medium | Hard | Overall |
|---|---:|---:|---:|---:|
| OpenAI Operator | 83.1% | 58.0% | 43.2% | 61.3% |
| Claude Computer Use 3.7, thinking disabled | 90.4% | 49.0% | 32.4% | 56.3% |
| SeeAct with GPT-4o | 60.2% | 25.2% | 8.1% | 30.7% |
| Browser Use with GPT-4o | 55.4% | 26.6% | 8.1% | 30.0% |
| Claude Computer Use 3.5 | 56.6% | 20.3% | 14.9% | 29.0% |
| Agent-E with GPT-4o | 49.4% | 26.6% | 6.8% | 28.0% |

Operator had the highest overall score in this experiment, while Claude Computer Use 3.7 had the highest easy-task score. Averaged over agents, the authors calculated a 31.6 percentage-point drop from easy to medium tasks and a further 15.4-point drop from medium to hard tasks. The result supports a difficulty effect within this evaluation. It should not be read as a current ranking of the products or model families named in the table.[1]

The paper's trajectory review identified recurring failure modes rather than a single dominant cause. These included overlooking numeric or temporal constraints, misunderstanding site-specific controls, insufficient exploration, looping through unproductive actions, hallucinating requirements, and relying too heavily on keyword search. Operator's reviewed failures often involved numeric or time constraints and niche website features. These are qualitative observations from the tested trajectories, not prevalence estimates for all browser agents.[1]

## Later leaderboard results and task revisions

The public leaderboard separates human and automatic evaluation. Its human table dated June 28, 2026 lists Yutori's Navigator n1.5 at 97.3 percent overall, with 100.0 percent on easy tasks, 96.5 percent on medium tasks, and 96.1 percent on hard tasks. Yutori states that the result received three independent reviews, additional quality assurance, and final verification by the benchmark authors.[11][12][14]

The associated o4-mini automatic table reports 87.9 percent for the same submission. Its row notes that a concise final response was included in the judge after being checked for factual consistency. That differs from the repository's default instruction to exclude final responses, so the automatic figure is not a clean instance of the paper's standard WebJudge input protocol.[3][13]

Neither number should be substituted into the paper's 2025 table without qualification. The benchmark maintainers replaced 36 invalid or CAPTCHA-affected tasks on November 3, 2025 and recorded additional task updates in November and December 2025 and in January and May 2026. They aim to preserve the website, task type, and reference length when replacing an item, but the new item is not literally the old web interaction. The repository also says simple tasks may be revised if performance saturates.[3]

As a result, a time series of leaderboard scores can mix at least three kinds of change: better agents, revised tasks, and revised judging or review procedures. Reproducible comparisons should identify the task snapshot and submission schema, preserve trajectories, and report human and automatic results separately.

## Comparison with related benchmarks

Online-Mind2Web occupies the live, open-web part of a larger browser-agent evaluation landscape:

| Benchmark | Environment | Main distinction |
|---|---|---|
| Mind2Web | Saved webpages and recorded demonstrations | Large offline dataset for learning and generalization across tasks, sites, and domains.[5] |
| Mind2Web-Live / WebCanvas | Live websites | Online execution with intermediate evaluation states; Online-Mind2Web imports 34 tasks from this lineage.[6] |
| [WebVoyager](https://aiwiki.ai/wiki/webvoyager) | Live websites | 643 tasks on 15 sites in the original release, with trajectory-based automatic evaluation.[7] |
| [WebArena](https://aiwiki.ai/wiki/webarena) | Self-hosted websites | Reproducible functional evaluation over 812 tasks in controlled site replicas.[8] |
| [VisualWebArena](https://aiwiki.ai/wiki/visualwebarena) | Self-hosted, visually grounded websites | Extends controlled web evaluation to tasks that require interpreting images and layouts.[9] |
| Online-Mind2Web | Maintained live websites | 300 tasks across 136 sites, human trajectory labels, and the WebJudge evaluation pipeline.[1][3] |

The designs address different tradeoffs. Self-hosted environments can be reset and inspected, but cover a smaller controlled web. Saved-page datasets support repeatable training and offline analysis, but cannot reproduce every behavior of a current site. Live benchmarks encounter real interfaces and failures, but their changing environment weakens strict repeatability. Scores should be compared within a benchmark and protocol before being used to claim broader agent superiority.

## Limitations

- **The test environment changes.** Layouts, inventory, prices, localization, authentication flows, cookie dialogs, and access policies can change between runs. Even an unchanged instruction can become easier, harder, or impossible.[1][3]
- **Task replacement complicates longitudinal comparison.** Maintaining 300 executable tasks is useful operationally, but a leaderboard row from one revision is not automatically comparable with a row from another.[3]
- **Human reference length is only a proxy for difficulty.** It does not isolate interface complexity, transient failures, or alternative valid paths.[1]
- **Automatic labels remain estimates.** WebJudge can generate flawed key points, omit relevant evidence, or apply completion criteria too strictly. Agreement with earlier human labels is not the same as correctness on a new agent distribution.[1]
- **System configurations are heterogeneous.** Agent scaffolds, models, step limits, screenshots, browsers, and hosted interfaces differ, so results do not isolate a single causal component.[1]
- **Coverage is broad but not universal.** The task set emphasizes executable consumer-web workflows and popular websites. It is not a representative sample of every language, region, accessibility condition, authenticated enterprise workflow, or browser task.[1]
- **Completion is not a safety evaluation.** The benchmark scores whether a requested state was achieved. It does not comprehensively test prompt injection resistance, privacy protection, transaction safety, authorization, or reliable abstention.
- **Full evaluation can be costly or restricted.** The maintainers note that privacy, legal constraints, and execution cost can make a complete run infeasible for some systems.[3]

## Availability and licensing

The official repository contains evaluation code, example trajectories, human labels, automatic-evaluation outputs, and submission instructions. Since May 2026, leaderboard submissions use an `online-mind2web-v2` schema in which each step bundles its action, thought, screenshot, and URL, reducing the risk that separately stored trajectory fields drift out of alignment. A task directory contains a `result.json` file and per-step images.[3]

The repository licenses its code under the MIT License and the Online-Mind2Web dataset under Creative Commons Attribution 4.0. The public Hugging Face dataset card identifies 300 English tasks and the fields `task_id`, `website`, `task_description`, and `reference_length`, but downloading the dataset files requires an authenticated user to accept access conditions and share contact information. "Publicly listed" therefore does not mean anonymous, unrestricted file access.[3][4]

WebJudge-7B has a separate Apache 2.0 model license and is based on Qwen2.5-VL-7B-Instruct.[10] Users should check all three artifacts independently because the code, task data, and model weights do not share one license.

## References

1. Xue, Tianci; Qi, Weijian; Shi, Tianneng; Song, Chan Hee; Gou, Boyu; Song, Dawn; Sun, Huan; Su, Yu. "An Illusion of Progress? Assessing the Current State of Web Agents." Conference on Language Modeling, 2025. https://openreview.net/forum?id=6jZi4HSs6o
2. Xue, Tianci et al. "An Illusion of Progress? Assessing the Current State of Web Agents." arXiv:2504.01382, revision history through October 8, 2025. https://arxiv.org/abs/2504.01382
3. OSU-NLP-Group. "Online-Mind2Web." Official repository, task update history, evaluation guidance, and submission schema. https://github.com/OSU-NLP-Group/Online-Mind2Web
4. OSU NLP Group. "Online-Mind2Web." Hugging Face dataset card and access conditions. https://huggingface.co/datasets/osunlp/Online-Mind2Web
5. Deng, Xiang et al. "Mind2Web: Towards a Generalist Agent for the Web." Advances in Neural Information Processing Systems 36, 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/5950bf290a1570ea401bf98882128160-Paper-Datasets_and_Benchmarks.pdf
6. Pan, Yichen et al. "WebCanvas: Benchmarking Web Agents in Online Environments." arXiv:2406.12373, 2024. https://arxiv.org/abs/2406.12373
7. He, Hongliang et al. "WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models." Proceedings of ACL, 2024. https://aclanthology.org/2024.acl-long.371/
8. Zhou, Shuyan et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents." International Conference on Learning Representations, 2024. https://proceedings.iclr.cc/paper_files/paper/2024/hash/4410c0711e9154a7a2d26f9b3816d1ef-Abstract-Conference.html
9. Koh, Jing Yu et al. "VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks." Proceedings of ACL, 2024. https://aclanthology.org/2024.acl-long.50/
10. OSU NLP Group. "WebJudge-7B." Hugging Face model card. https://huggingface.co/osunlp/WebJudge-7B
11. OSU NLP Group. "Online-Mind2Web Leaderboard." Hugging Face Space. https://huggingface.co/spaces/osunlp/Online_Mind2Web_Leaderboard
12. OSU NLP Group. "Online-Mind2Web Human Leaderboard Data." Hugging Face Space source file, revision cc0a52c. https://huggingface.co/spaces/osunlp/Online_Mind2Web_Leaderboard/blob/cc0a52ccefa27debad265d3e9f33b52e378573d7/human_Mind2Web-Online%20-%20Leaderboard_data.csv
13. OSU NLP Group. "Online-Mind2Web o4-mini Automatic Leaderboard Data." Hugging Face Space source file, revision cc0a52c. https://huggingface.co/spaces/osunlp/Online_Mind2Web_Leaderboard/blob/cc0a52ccefa27debad265d3e9f33b52e378573d7/auto_o4-mini_Mind2Web-Online%20-%20Leaderboard_data.csv
14. Yutori. "Navigator n1.5 solves Online-Mind2Web with 97.3% success." Product changelog, July 2, 2026. https://yutori.com/changelog