WindowsWorld

WindowsWorld is a process-aware benchmark for evaluating autonomous computer-use agents on professional workflows that cross several Windows applications. Jinchao Li and colleagues introduced it in 2026, and the final paper appeared in Findings of the Association for Computational Linguistics: ACL 2026. The benchmark contains 181 tasks across 17 desktop applications. Of those tasks, 141 involve at least two applications, and each feasible task has state-based checkpoints that measure partial progress as well as final completion.^[1]

WindowsWorld runs agents in controlled Windows virtual machines and asks them to operate ordinary graphical interfaces. Its central distinction is therefore narrower than "real-world computer use" in general: it tests whether a particular model, agent scaffold, observation format, and action interface can maintain a workflow across Windows programs under fixed step limits. In the paper's experiments, the highest reported overall final-completion score was 20.44%, while the same configuration satisfied 50.32% of intermediate checkpoints. These are results from the paper's named model snapshots and evaluation harness, not a continuously updated leaderboard.^[1]

Scope and task design

The benchmark organizes tasks by difficulty and by one of 16 occupational personas in five broad domains. Examples include accountants, HR specialists, designers, software developers, and general users. The personas guide task generation, but they are not a sample of observed employee activity. WindowsWorld measures the authors' curated representation of professional workflows rather than the prevalence of those workflows in workplaces.^[1]

The four levels have different goals and execution budgets:

Scroll sideways for more →

L4 tasks contain an obstacle such as an invalid URL, a missing file, or required authentication. They test failure recognition rather than successful GUI manipulation. This changes the meaning of the aggregate final score because a correct L4 outcome is a refusal, while a correct L1-L3 outcome is task completion.^[1]

The task set contains 40 one-application tasks, 43 two-application tasks, 86 three-application tasks, 10 four-application tasks, and 2 five-application tasks. The mean is about 2.4 applications per task, and 77.9% of tasks are multi-application. L2 and L3 together account for 71.8% of the benchmark.^[1][2]

The paper groups the 17 applications as follows, combining productivity, communication, system, multimedia, and programming software. Counts indicate how many tasks involve an application, so a multi-application task contributes to several counts.

Scroll sideways for more →

The authors also estimated the minimum actions an expert needed for each feasible level. The reported averages were 9.67 actions for L1, 18.13 for L2, and 27.81 for L3. These are action-count references, not a human success-rate baseline.^[1]

Task construction

WindowsWorld uses a human-in-the-loop generation pipeline rather than writing every task manually. A generator based on DeepSeek-V3.2 receives a persona, routine descriptions, and application dependencies. It can search an allowlist of open websites when generating web-dependent instructions.^[1]

A four-part refiner then processes candidate tasks:

1. A semantic deduplicator removes near-duplicate instructions above a cosine-similarity threshold of 0.85.
2. A validity auditor checks referenced URLs and files.
3. A dependency reasoner converts procedural assumptions into environment requirements.
4. A metric refiner turns task requirements into checkpoints intended to describe required states instead of a single prescribed click sequence.

Four human annotators reviewed generated tasks. According to the paper, they rejected ambiguous instructions, subjective criteria, and tasks that depended on unavailable software or inaccessible services. An environment generator then created required files such as spreadsheets, documents, and scripts. It also merged compatible resources for tasks assigned to the same persona.^[1]

This method separates task breadth from final quality control, but it does not make the tasks direct observations of professional work. The occupational framing, generated files, and task wording all depend on the pipeline's prompts and the reviewers' decisions.

Environment and agent interface

The paper describes controlled Windows virtual machines in which agents use GUI operations without internal application APIs or privileged system calls. The public repository documents Windows 10, Windows 11, Windows Server 2022, and Windows Server 2025 images through VMware Workstation Pro. It requires vmrun on the host and Python 3.11 or later for the released runner.^[1][2]

The evaluation supports three observation settings for general-purpose models:

Scroll sideways for more →

The release supports free-form PyAutoGUI actions and the 13-action interface inherited from OSWorld. The paper mainly evaluates general-purpose models with PyAutoGUI. The UiPath baseline instead uses the 13-action interface. Agent-S3 and UiPath configurations also use UI-TARS-1.5-7B as a grounding model, so their scores cannot be attributed to the named foundation model alone.^[1][2]

The repository publishes code, the 181-record benchmark.json, an installation guide, and a virtual-machine image link under an Apache-2.0 repository license. The README states that substantial parts of the evaluation framework derive from OSWorld and remain subject to that license.^[2][3]

Process-aware scoring

For a trajectory (\tau), a set of checkpoints (C), and terminal state (s_T), the paper defines:

$$S_{\mathrm{int}} = \frac{1}{|C|} \sum_{k=1}^{|C|} J(\tau,c_k), \qquad S_{\mathrm{final}} = J(\tau,s_T).$$

The judge (J), implemented with Qwen3-VL-Plus, returns a binary decision for each checkpoint and for the final state. (S_{\mathrm{int}}) is reported only for L1-L3 tasks, so L4 records do not contribute to that metric. (S_{\mathrm{final}}) covers all four levels.^[1]

Checkpoint descriptions are intended to be state based. The construction process removes instructions tied to one action, such as clicking a named button, and keeps required semantic states, such as having the target file open. Human reviewers then check that the checkpoint is necessary without requiring one exact path. This permits shortcuts, menus, and keyboard commands that reach the same state.^[1]

The authors compared the automated judge with the consensus of two human annotators on 100 stratified tasks containing 518 checkpoints:

Scroll sideways for more →

The paper's error analysis found false positives when a trajectory appeared to reach a state that was later hidden or changed. False negatives occurred when a target state appeared only briefly and was no longer visible in a later screenshot. The scores therefore have measured agreement with human annotations, but the judge is not error free.^[1]

Reported results

The paper evaluated five general-purpose model snapshots under screenshot, Set-of-Marks, and hybrid observations. The following rows use the hybrid setting so the base-model comparison has a common observation format:

Scroll sideways for more →

The exact observation choice matters. For example, GPT-5.2 reached a higher final score of 1.78% with screenshot-only input, and its Set-of-Marks configuration correctly rejected 25% of L4 tasks. The same configuration did not produce the model's best overall feasible-task score. A single row should therefore not be treated as a context-free ranking of the underlying frontier models.^[1]

The paper also reported three scaffolded agent configurations with screenshot input:

Scroll sideways for more →

The released evaluator treats an L4 run as correct when the action list is empty or its last action indicates FAIL. It does not evaluate whether the agent supplied a correct explanation for that refusal. The paper notes the resulting interpretation problem: UiPath reached 50% on L4 partly because it often declared tasks incomplete, even though it performed poorly on feasible tasks.^[1][5]

Cross-application effect

Longer tasks are an obvious confounder when comparing L1 and L2. The authors therefore selected subsets with similar estimated minimum action counts, 10.92 for L1 and 11.26 for L2. On those subsets, the reported intermediate score fell from 65.74% to 35.14%, while final completion fell from 46.15% to 14.29%. Within that experiment, the application boundary was associated with a large score drop even after approximately matching task length.^[1]

The benchmark also separates partial progress from completion. On L3 hybrid tasks, model checkpoint scores ranged from 2.04% to 38.63%, while final scores stayed below 15%. This distinction is useful for diagnosing where a trajectory stopped, but satisfying half the checkpoints does not mean that half the user's intended result was delivered. Checkpoints are discrete states chosen by the benchmark authors, not a general measure of economic value or time saved.^[1]

Relation to other benchmarks

Windows Agent Arena and OSWorld supplied the closest Windows evaluation infrastructure before WindowsWorld. OSWorld introduced execution-based evaluation over 369 tasks across Ubuntu, Windows, and macOS, including file operations and some multi-application workflows. Windows Agent Arena adapted that infrastructure to Windows, with more than 150 tasks and parallel execution in Azure.^[6][7]

WindowsWorld's paper reports that 27.4% of the 49-task Windows subset it analyzed from OSWorld was multi-application, compared with 77.9% in WindowsWorld. Its other distinguishing feature is checkpoint scoring. Earlier final-state evaluators can recognize multiple valid outcomes, but they generally do not assign progress credit for required intermediate states.^[1][6][7]

Process information was not unique to WindowsWorld. ProBench, published at AAAI 2026, evaluates more than 200 mobile GUI tasks and includes process-related tasks through a Process Provider. WindowsWorld differs in platform and in its concentration on cross-application desktop work.^[8]

Later 2026 benchmarks expanded the same research direction:

Scroll sideways for more →

As of July 2026, WindowsWorld is best understood as one benchmark with a specific combination of Windows virtual machines, cross-application task composition, partial-progress checks, and infeasible instructions. Results from the later suites are not directly comparable because their tasks, environments, step limits, judges, and scoring weights differ.

Limitations and reproducibility

The paper and release support several boundaries on interpretation:

• Task count and coverage: 181 tasks provide many workflows but only 12 infeasible cases. The 16 personas do not represent every occupation, organization, language, or software stack.^[1]
• Generated task distribution: The benchmark uses model-generated instructions and synthesized input files followed by human filtering. It does not estimate how often these tasks occur in actual employment.^[1]
• No human completion baseline: Expert minimum action counts provide a horizon estimate, but the paper does not report a human success rate on the full benchmark.^[1]
• Judge dependence: Intermediate scores require complete trajectories, manually reviewed checkpoints, and a vision-language judge. The authors identify this as a limit on scaling the method for online reinforcement learning.^[1]
• Language and interface coverage: The paper compares Chinese and English instructions, but the environment is optimized for a single-language operating-system interface. It does not establish multilingual Windows robustness.^[1]
• Tool boundary: The published evaluation does not test Model Context Protocol tools or application APIs. It measures GUI interaction through its supported action spaces.^[1]
• Point estimates: The main result table reports percentages for named configurations without run-to-run confidence intervals. Repeated attempts, model updates, or different inference settings may change the scores.^[1]

The public release also requires version-specific reporting. At the repository head checked on July 28, 2026, commit fbccd464f94fec9e284e139f97bf96d0b192f580, benchmark.json contained all 181 records. Fifty-five records retained review_status set to pending, and 21 had validity_info.is_valid set to false. The default runner loads every record from the supplied JSON and does not filter those two fields. The README does not document a filtering rule for them.^[2][3][4]

Those metadata do not by themselves establish that a task is unusable, but they prevent readers from assuming that every distributed record carries the same recorded review state. A reproducible result should state the repository commit, virtual-machine image, task filtering rule, model snapshot, agent scaffold, observation and action modes, inference settings, step budgets, and number of attempts.

See also

• AI benchmark
• Agentic AI

References

1. ^Jinchao Li, Yunxin Li, Chenrui Zhao, Zhenran Xu, Baotian Hu, and Min Zhang. "WindowsWorld: A Process-Centric Benchmark of Autonomous GUI Agents in Professional Cross-Application Environments." *Findings of the Association for Computational Linguistics: ACL 2026*, pages 15262-15280, July 2026. doi.org/...2026.findings-acl.750
2. ^HITsz-TMG. "WindowsWorld" repository, commit `fbccd464f94fec9e284e139f97bf96d0b192f580`, accessed July 28, 2026. github.com/...d464f94fec9e284e139f97bf96d0b192f580
3. ^HITsz-TMG. "`benchmark.json`," WindowsWorld repository, commit `fbccd464f94fec9e284e139f97bf96d0b192f580`, accessed July 28, 2026. raw.githubusercontent.com/...benchmark.json
4. ^HITsz-TMG. "`main.py`," WindowsWorld repository, commit `fbccd464f94fec9e284e139f97bf96d0b192f580`, accessed July 28, 2026. github.com/...main.py
5. ^HITsz-TMG. "`hf_run.py`," WindowsWorld repository, commit `fbccd464f94fec9e284e139f97bf96d0b192f580`, accessed July 28, 2026. github.com/...hf_run.py
6. ^Tianbao Xie et al. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." *Advances in Neural Information Processing Systems 37*, 2024. proceedings.neurips.cc/...ets_and_Benchmarks_Track
7. ^Rogerio Bonatti et al. "Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale." arXiv:2409.08264, September 2024. arxiv.org/...2409.08264
8. ^Leyang Yang, Ziwei Wang, Xiaoxuan Tang, Sheng Zhou, Dajun Chen, Wei Jiang, and Yong Li. "ProBench: Benchmarking GUI Agents with Accurate Process Information." *Proceedings of the AAAI Conference on Artificial Intelligence*, 40(32), 27547-27555, 2026. doi.org/...aaai.v40i32.39974
9. ^Kean Shi et al. "SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?" arXiv:2605.15777, May 2026. arxiv.org/...2605.15777
10. ^Liya Zhu et al. "Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic Tasks in Real-World Professional Fields." arXiv:2606.11042, June 2026. arxiv.org/...2606.11042
11. ^Mengqi Yuan et al. "OSWorld 2.0: Benchmarking Computer-Use Agents on Long-Horizon Real-World Tasks." arXiv:2606.29537, June 2026. arxiv.org/...2606.29537
12. ^XLANG Lab. "OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks." Project website, accessed July 28, 2026. osworld-v2.xlang.ai