Mage-VL-Cowork

Mage-VL-Cowork is a 4-billion-parameter computer-use agent model released by the Microsoft Mage Team at Microsoft on 30 October 2026. It is a single vision-language model policy that works a desktop through two action channels: screenshot-grounded mouse and keyboard actions (a computer_use tool) and a persistent shell and Python environment (a bash tool). The model decides at each turn which channel to use, without an external orchestrator or router.^[1][2] Microsoft reports a score of 66.0% on OSWorld-Verified under a 50-step budget,^[1][2][3] which it describes as the best result among 4B models.^[1][2] The weights are published on Hugging Face under the Apache-2.0 license, alongside a 30-page technical report titled Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use from Evolved Harnesses.^[1][2][5]

The model was trained from an intermediate checkpoint of Mage-VL, Microsoft's 4B codec-native video and image understanding model, through a five-stage "specialize-then-unify" curriculum of midtraining, supervised fine-tuning and online reinforcement learning.^[1][2] Its training data came from Mage-CUA-Forge, a task-synthesis pipeline that Microsoft says generated 293,806 desktop and web tasks across 6,915 atomic skills while cutting recurring per-task generation cost by 82.6%.^[2] All benchmark figures on this page are Microsoft's own results unless stated otherwise. As of 31 October 2026, a report on the German financial-news site ad-hoc-news.de said no independent third-party evaluation had been published and that the model was not listed on the official OSWorld leaderboard.^[13]

Overview

Background: the Mage family

Mage is a family of compact multimodal models from Microsoft, each built at a fixed 4B-parameter budget. The project's GitHub repository describes it as "a family of lightweight, research-friendly multimodal models" intended for controlled experiments, post-training research and vertical-domain applications on modest compute.^[6] As of 1 November 2026 the family has three branches:^[6]

Mage-VL pairs a vision encoder trained from scratch, Mage-ViT, with the Qwen3-4B-Instruct-2507 language model, which its model card calls "the only pretrained component". Mage-ViT follows the structure of video codecs: it keeps every patch of anchor (I) frames but only the predicted (P) frame patches where the codec spends bits, which Microsoft says cuts visual tokens by over 75% and gives up to a 3.5x wall-clock inference speedup over uniform frame sampling.^[8][9] A lightweight "cognition gate" lets the same checkpoint stay silent on routine video and respond only when an event completes. Microsoft reports that Mage-VL roughly matches Qwen3-VL-4B on static image benchmarks and beats it on every reported video and temporal-grounding benchmark.^[8][9] The standalone encoder, Mage-ViT, is released separately under the MIT license as a pretrained-only checkpoint of about 316 million parameters.^[10] The Mage-VL arXiv paper was submitted on 27 July 2026 with 23 authors.^[9]

Release and availability

The Mage-VL-Cowork model repository on Hugging Face was created on 30 October 2026 at 12:13 UTC.^[5] Commits adding the technical report PDF, the model link and an OSWorld-Verified reproduction suite to the microsoft/Mage GitHub repository were made the same day.^[7] As of 1 November 2026, the Hugging Face API listed 233 downloads and 89 likes for the model, and the microsoft/Mage repository, which hosts code for the whole family, had 1,734 stars and 175 forks.^[5][7] The technical report is hosted as a PDF in the GitHub repository; its suggested citation lists it as an "arXiv preprint" without an identifier.^[1][6] The report credits 22 contributors, with Xiaoyi Zhang as project lead and Xiaoyi Zhang and Haoqing Wang as equal contributors.^[2]

The model uses a custom architecture defined in bundled Python files, so it must be loaded with trust_remote_code=True in transformers or SGLang. The model card warns against assuming weight compatibility with standard Qwen3-VL classes.^[1] The model card points to a prebuilt SGLang inference container and recommends sampling at temperature 0.7, top-p 0.8 and a 4,096-token output budget, the configuration used for the 66.0% result.^[1] The card also gives safety guidance: run the agent only in isolated, disposable sandboxes such as Docker containers or snapshot-reverted virtual machines; require human confirmation before irreversible actions such as financial transactions or privilege changes; keep personal credentials out of the workspace; and restrict outbound network access to reduce indirect prompt-injection risk.^[1]

Architecture

According to the released config.json, the language model is a 36-layer Qwen3 decoder with a hidden size of 2,560, 32 attention heads and 8 key-value heads, a vocabulary of 151,936 tokens and a maximum position embedding of 262,144. The vision tower has 24 layers, a hidden size of 1,024, 16 attention heads, a patch size of 16 and a spatial merge size of 2, and projects into the 2,560-dimensional text space.^[4] The model card describes the vision side as Mage-ViT with a 16x16 patch grid and 3D rotary position encoding.^[1]

Action channels and output format

The model is trained against a fixed system prompt that exposes two tools.^[1]

Each turn has three parts: a <think> reasoning block, a one-sentence Action: summary, and one or more <tool_call> blocks. The model card's example shows the model choosing to edit an .xlsx file with openpyxl from the shell instead of opening LibreOffice Calc.^[1] Microsoft says the policy learns to use the GUI for visual layouts, canvas editing and live in-memory application state, and the shell for structured edits to spreadsheets, documents and presentations and for checking results against the files on disk.^[1][2]

Working memory

To fit long tasks into a 32K context, the model uses an asymmetric sliding window. The three most recent screenshots and four complete assistant responses stay in full, so the model can compare before-and-after screens and notice clicks that did nothing. Earlier turns are folded into one-line Action: summaries placed after the system prompt.^[1][2] The report says an earlier design with ten text turns and a single screenshot often fell into repeated click loops after a slightly off-target click.^[2]

Training data

Teacher-harness evolution

The training trajectories come from a frontier teacher model running inside an agent harness, and Microsoft argues that the harness is not a neutral container: its tool interfaces, context handling and guidance directly shape how a fixed teacher model acts. The report defines the harness as four parts: the execution scaffold, the tool interfaces, the system prompt and the preinstalled terminal environment. An authoring agent revises these on a development set to improve the trade-off between success and cost.^[2] Changes included removing xdotool and verbose accessibility-tree dumps, which duplicated the GUI action route, and preinstalling libraries such as python-pptx, openpyxl, pandas, pypdf and pillow.^[1][2] The evolution produced specialized GUI, terminal and hybrid harnesses, and the same synthesized tasks were rolled out under each to supply different training stages.^[2]

In an ablation on 672 held-out multi-application tasks with GPT-5.4 as the fixed teacher, Microsoft compared a pruned GUI-plus-coding harness with one that also exposed an accessibility channel:^[2]

A 2.4-point gap for the teacher became a 12.5-point gap for the distilled students. The report's explanation is that a frontier teacher can absorb a redundant channel by taking extra steps, while a small student is much more sensitive to ambiguous channel routing.^[2] The approach is a form of knowledge distillation from teacher trajectories.

Mage-CUA-Forge

Mage-CUA-Forge is the task and environment synthesis pipeline. Instead of running an open-ended agent loop to build each task, an authoring agent builds and debugs one reusable generation workflow per atomic skill, plus application-specific "preparers" that create input artifacts such as a Calc workbook or a Thunderbird profile with a populated mail store. Each task package contains a user instruction, an environment setup script and an evaluation rubric, and consistency gates check packages before they are accepted.^[2] An exploration agent built the initial skill taxonomy by traversing desktop applications, and the taxonomy was expanded over three rounds based on where the student model was weak.^[2] For web and cross-application tasks, the pipeline uses synthesized InfiniteWeb sites and self-hosted website replicas extended from WebHarbor, which run on Flask and SQLite and reset from a seed database before every episode.^[2] Task completion was judged by GPT-5.6 Terra (gpt-5.6-terra) against the rubric, and step-level reasoning text was synthesized with GPT-5.4-mini.^[2]

Using the same generation model for both approaches, Microsoft reports that the workflow-first method cut recurring generation cost from 1,276K tokens and $0.906 per task (agent-first construction) to 85K tokens and $0.158 per task, reductions of 93.3% in tokens and 82.6% in cost. The comparison excludes the one-time cost of authoring and debugging the workflows.^[2]

Training curriculum

The curriculum starts from an intermediate 4B Mage-VL checkpoint. The report says this checkpoint was weak at visual grounding, chain-of-thought reasoning and structured tool calling, tending to give terse direct answers. The curriculum builds a GUI channel and a coding channel separately before teaching one policy to combine them.^[2] Midtraining and SFT used two epochs over 32,768-token packed sequences.^[2]

Sources for the table: model card and technical report.^[1][2] By phase, midtraining took 15.2% of nominal compute, the three SFT stages 51.2% and RL 33.6%. The estimate covers language-model training only and leaves out base pretraining and vision-encoder computation.^[2]

Stage 1 used a minimal click-only system prompt for grounding data, because training click-only examples under the full agent prompt made the policy click unconditionally. It removed duplicate queries about the same on-screen element from packed sequences so the model could not copy earlier coordinates, and it masked the interior of <think> blocks so verbose reasoning would not dominate the gradients.^[2] During Stage 1, grounding accuracy rose from 30.6% to 56.9% on OSWorld-G and from 78.8% to 87.2% on ScreenSpot-v2.^[1][2] The checkpoint after Stage 4 is called Mage-VL-Cowork-SFT.^[2]

Stage 5 follows the ScaleCUA recipe: frontier sampling weights tasks by prior success rate. Each optimizer update admits 96 task groups of eight episodes (768 slots), policy lag is capped at three updates, and truncated importance sampling caps weights at 2.0.^[1][2] ScaleCUA is a separate arXiv paper from July 2026 that reports 68.7% on OSWorld.^[12]

RL infrastructure

Online RL ran on a "Rollout-as-a-Service" system that separates desktop execution, inference and training. It combines PoLar, the Slime training framework, SGLang for inference, Azure Relay and Azure Virtual Machine Scale Sets.^[2] The reference deployment used 32 NVIDIA B200 GPUs (24 for training and 8 for inference) and 1,120 desktop sandbox slots across 70 CPU hosts, each a 64-vCPU Standard_D64d_v5 machine.^[2] Some desktop episodes ran longer than 40 minutes, which is why rollouts were made asynchronous. In the profiled window, mean update time was 570.3 seconds and mean policy lag was 2.71 updates.^[2]

Benchmark results

OSWorld-Verified

Microsoft evaluated the final model on 361 OSWorld-Verified tasks, excluding Google Drive tasks, with a 50-step budget at 1920x1080 resolution. The headline 66.0% (66.04%) is the average of two full runs scoring 65.33% and 66.75%.^[1][2] The table below reproduces Microsoft's comparison as published. The report says baseline scores and step budgets are "shown as reported" by their original sources; only GPT-5.4-mini was run by Microsoft, in a single run on the same 361 tasks.^[2]

Blank cells are values Microsoft's table does not give. Source: technical report Table 2 and model card.^[1][2]

The 87.5% OS-category score is the highest in Microsoft's table, and the 62.7% Workflow score is 14.6 points above ScaleCUA-Qwen3.5-9B. Overall, though, the 9B ScaleCUA model (68.7%) scores higher than Mage-VL-Cowork (66.0%), so the "best" claim applies to the 4B size class.^[2] Microsoft says the model outperforms open-source agents up to 7.5 to 18 times larger, naming Muse Glimmer-30B, Qwen3.6-27B, EvoCUA-32B and OpenCUA-72B.^[2]

Stage-wise progression and efficiency

Source: model card and technical report.^[1][2]

Hybrid SFT added 10.28 points over GUI SFT and cut model calls by 14.9%, but raised mean output tokens. Online RL then added 10.16 points over Hybrid SFT while cutting mean model calls by 13.0% and both output and think tokens by about 25.4%.^[1][2] Microsoft notes that the final model uses about as many turns per task as GPT-5.4-mini (18.77 versus 18.24) with 55.6% fewer output tokens.^[1]

WebVoyager-Static

For browser tasks, Microsoft built WebVoyager-Static, an offline version of WebVoyager that reconstructs all 643 tasks across its 15 websites on resettable self-hosted replicas. Grading checks both the agent's page-visit trace and the backend SQLite state, instead of relying only on a screenshot-based judge.^[1][2] The SFT checkpoint (before RL) was evaluated:^[2]

Mage-VL-Cowork-SFT solved 377 of 643 tasks and placed second, behind the browser-only specialist Fara1.5-4B.^[2] Microsoft argues that the 12.8 to 33.1 point drops from published live scores show score inflation in live web evaluation. It names three failure modes that it says screenshot-only judges miss: answering from search snippets or memory without opening the detail page, failing to apply secondary filters, and declaring success on forms that never committed to the backend.^[2]

Side-by-side demonstrations

The model card includes two timed comparisons against Qwen3.6-27B running GUI-only in thinking mode. On a LibreOffice Calc task (merge ranges, add a chart, save as .xlsx), Mage-VL-Cowork finished in 39 seconds and 6 steps by scripting the workbook in Python, against 9 minutes 56 seconds and 39 steps for Qwen3.6-27B. On a GIMP task, Mage-VL-Cowork finished in 1 minute 58 seconds and 15 steps, editing in the GUI and then checking the file from the terminal; Qwen3.6-27B stopped after 7 minutes 2 seconds with a file that failed verification.^[1] These are single demonstrations chosen by Microsoft, not benchmark results.

Caveats and reception

• Self-reported results. Every score above comes from Microsoft. The comparison mixes baselines run under different step budgets (30, 50 or 100 steps) and scores copied from other publications, and the OSWorld-Verified evaluation leaves out Google Drive tasks.^[2] A 31 October 2026 report on ad-hoc-news.de (editorial team of boerse-global.de) said no independent third-party test was available yet and that the model did not appear on the official OSWorld leaderboard.^[13]
• Comparator figures vary by source. Microsoft's table lists Qwen3.6-27B at 63.9% on OSWorld-Verified. Meta's Muse Glimmer model card listed Qwen3.6-27B (thinking mode) at 75.6% on the same benchmark name, and Muse Glimmer-30B at 65.9%.^[2][14] Under Meta's figure, Qwen3.6-27B would rank above Mage-VL-Cowork.
• Teacher dependence. The training data relies on frontier models: an unnamed frontier teacher for the trajectories (the report names GPT-5.4 only as the fixed teacher in its harness ablation, and its harness-evolution figure includes a step that switches the teacher's underlying model), OpenAI's GPT-5.6 Terra as the rubric judge and task generator, and GPT-5.4-mini for reasoning annotation.^[2]
• Security. Because the model can run arbitrary shell commands, Microsoft says it is designed and evaluated for isolated, disposable sandboxes and should always be run inside an isolated container or VM.^[1]

Continue reading

• OOSWorldOSWorld is a benchmark for evaluating multimodal AI agents on open-ended tasks performed inside real computer environments.Mentioned 12 times above
• CComputer useComputer use is the capability of artificial intelligence models to control computers by viewing screens and performing mouse and keyboard actions, much like a human user would.Mentioned 6 times above
• TTerminal-BenchTerminal-Bench is an open benchmark for evaluating AI agents on complex, real-world tasks performed through command-line terminal interfaces.Listed in See also
• SSWE-benchSWE-bench is an execution-based benchmark for evaluating whether a language-model system can resolve real software issues.Listed in See also
• SSmall language modelA small language model (SLM) is a compact large language model, typically under about 10 billion parameters, built to run efficiently on a single consumer GPU, a laptop, or a smartphone rather than in a data…Listed in See also
• Phi-4Phi-4 is a 14-billion-parameter small language model developed by Microsoft Research and released in December 2024, designed to match or beat models several times its size on reasoning tasks by training…Listed in See also

More to read

• Microsoft
• Computer-use agent
• Vision language model
• Qwen3
• Qwen3.6
• Qwen3.5

Topics:AI AgentsMicrosoftMultimodal AIOpen Source AI

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38Microsoft Mage Team, *microsoft/Mage-VL-Cowork model card*, Hugging Face, accessed 1 November 2026. huggingface.co/...Mage-VL-Cowork
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49 ^50 ^51 ^52 ^53 ^54 ^55 ^56Microsoft Mage Team (Xiaoyi Zhang, Haoqing Wang et al.), *Mage-VL-Cowork: A Full Training Recipe to Learn Hybrid Computer Use from Evolved Harnesses*, technical report (PDF), October 2026. github.com/...mage_vl_cowork_tech_report.pdf
3. ^Microsoft Mage Team, *Mage-VL-Cowork project page*, accessed 1 November 2026. microsoft.github.io/...cowork
4. ^1 ^2 ^3 ^4 ^5Microsoft, *Mage-VL-Cowork config.json*, Hugging Face, accessed 1 November 2026. huggingface.co/...config.json
5. ^1 ^2 ^3 ^4 ^5 ^6Hugging Face, *API metadata for microsoft/Mage-VL-Cowork* (createdAt, downloads, likes, safetensors parameter count), accessed 1 November 2026. huggingface.co/...Mage-VL-Cowork
6. ^1 ^2 ^3 ^4 ^5 ^6Microsoft Mage Team, *Mage: A Lightweight, Research-Friendly Multimodal Model Family*, GitHub repository README, accessed 1 November 2026. github.com/...Mage
7. ^1 ^2GitHub, *API metadata and commit history for microsoft/Mage*, accessed 1 November 2026. api.github.com/...Mage
8. ^1 ^2Microsoft Mage Team, *microsoft/Mage-VL model card*, Hugging Face, accessed 1 November 2026. huggingface.co/...Mage-VL
9. ^1 ^2 ^3 ^4Senqiao Yang, Kaichen Zhang, Zhaoyang Jia et al., *Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model*, arXiv:2607.24904, 27 July 2026. arxiv.org/...2607.24904
10. ^Microsoft Mage Team, *microsoft/Mage-ViT model card*, Hugging Face, accessed 1 November 2026. huggingface.co/...Mage-ViT
11. ^Xinjie Zhang, Peng Zhang, Shicheng Zheng et al., *Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing*, arXiv:2607.19064, 21 July 2026. arxiv.org/...2607.19064
12. ^Bowen Lv, Xiao Liu, Yanyu Ren et al., *ScaleCUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RL*, arXiv:2607.11185, 13 July 2026. arxiv.org/...2607.11185
13. ^1 ^2Redaktion boerse-global.de, *Mage-VL-Cowork: Microsoft veröffentlicht Desktop-KI mit 4 Milliarden Parametern*, ad-hoc-news.de, 31 October 2026. ad-hoc-news.de/...70439398
14. ^Meta, *Muse Glimmer model card* (benchmark table), Hugging Face, accessed 1 November 2026. huggingface.co/...Muse-Glimmer-30B

Show all 14 references