BlenderWorld

BlenderWorld is a benchmark for visual AI agents that build editable 3D scenes in Blender, the open-source 3D creation suite, from text briefs, reference images, reference videos and structured specification files. It was released on 14 October 2026 by researchers from Alibaba's Qwen team and four universities, with an arXiv paper, an MIT-licensed evaluation toolkit on GitHub, and dataset copies on Hugging Face and ModelScope.^[1][2][7] The benchmark has 223 tasks scored against 2,651 task-specific binary criteria plus eight visual-quality ratings. Agents are judged on the native .blend project they deliver, not on a single image, and an evaluator agent re-renders each scene from eight views of its own choosing.^[1][2] In the authors' own runs of 13 agents, GPT-6-Astra had the highest overall score (79.9 out of 100), followed by Opus-5 (73.0) and GPT-5.6-Sol (67.7). No agent fully satisfied every criterion on more than 42.2% of tasks.^[1]

Background

The paper places BlenderWorld among benchmarks that test whether multimodal large language model agents can produce real deliverables in software, and extends that idea from 2D design work (webpages, posters, slides) to 3D modeling and scene creation.^[1] The authors name three problems with evaluating agents on practical 3D tasks: the environment must support creation, editing and inspection; real requests vary widely in their references and goals; and realistic requests often have no ground-truth 3D asset and admit several valid answers, so a single rendered view is not enough evidence.^[1]

They chose Blender because it supports all three capabilities through Python scripting and Model Context Protocol (MCP) interfaces.^[1] According to the paper, earlier Blender-based benchmarks cover narrower task classes: 3DCodeBench and 3DHarnessBench focus on individual objects, BlenderBench has a small set of camera and scene-editing tasks, and SceneActBench covers broader scenes but with predefined geometric objectives.^[1]

Release

The arXiv preprint, "BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References" (arXiv:2610.23986, category cs.CV), was submitted on 14 October 2026.^[1] The GitHub repository QwenLM/BlenderWorld was created the same day, and the Hugging Face dataset Qwen/BlenderWorld was published on 14 October and last modified on 15 October.^[2][5] IT Home (IT之家) reported on 17 October that the Qwen team had open-sourced BlenderWorld on 14 October under the MIT license, with code and data on GitHub, Hugging Face and ModelScope.^[7] On 23 October the Qwen Developers account on X said "we open-sourced BlenderWorld last week" and described it as "a benchmark with 223 tasks where agents use Blender to build editable 3D scenes from text, images or video references."^[6]

The paper lists 15 authors: Xinyu Fang, Yuchi Wang, Weihao Yu, Rongyao Fang, Zhaohai Li, Zhixun Li, Tiancheng Gu, Xilin Wei, Yuhang Zang, Hongsheng Li, Dahua Lin, Guofeng Zhang, Shuai Bai, Zhibo Yang and Dayiheng Liu. Their listed affiliations are Zhejiang University; Alibaba Token Hub, Alibaba Group; The Chinese University of Hong Kong; Fudan University; and the National University of Singapore. Dahua Lin, Guofeng Zhang and Zhibo Yang are marked with a dagger, which the paper does not define; the listed contact addresses are those of Xinyu Fang and Zhibo Yang.^[1]

The GitHub repository's code is released under the MIT License, and the Hugging Face dataset card also lists MIT. The README of the Hugging Face Harbor package states that its MIT LICENSE covers the adapter and evaluator code.^[2][5]

Task design

Each task gives the agent a natural-language request, optional image or video references, and optional specification files. The agent works in Blender through its Python API and MCP tools (scene inspection, Python execution, asset import and rendering), using rendered views and scene queries to revise geometry, materials, lighting and animation. It must deliver an editable Blender project, plus the renders, animations or other files the brief asks for.^[1]

Each visual reference is assigned one of three roles:^[1]

Briefs are set in realistic application scenarios; the paper's example is that construction reports stress structural relationships while product presentations stress visibility and appearance. Detailed tasks can carry Markdown, YAML or JSON attachments with dimensions, material lists, camera settings or asset budgets.^[1] Briefs were produced with help from a multimodal model and then checked twice. A structural check confirms that metadata, files and attachments are valid and that the brief asks for both renders and an editable project. A semantic review, done by a different model from the task generator, checks image-text alignment, constraint consistency and feasibility. Failing tasks go back for revision.^[1]

Rubrics were generated from the validated briefs as binary pass/fail criteria. Each criterion has one of 13 categories (presence, count, geometry, scale, layout, space_layout, material, color, lighting, composition, physics, function, holistic), a weight of 3, 2 or 1 for major, secondary or detail requirements, and an evidence type: render for visual properties or scene_summary for project and file inspection.^[1] According to the appendix, GPT-5.6-Sol drafted and edited the rubrics, and pilot judgments of its own outputs were used to flag ambiguity. At least two reviewer models (drawn from GPT-5.6-Sol, Gemini 3.1 Pro Preview and Claude Opus 4.8) checked each candidate independently, and human review supplemented the process before the rubrics were frozen.^[1]

Dataset statistics

The 223 tasks fall into 9 main categories and 22 subcategories and serve 16 application scenarios.^[1][2]

Of the 78 text tasks, 22 include specification files. Across the benchmark, 59 tasks (26.5%) have specification attachments: 28 Markdown, 20 YAML and 11 JSON files.^[1]

Category counts are from the project website's data file.^[4] The largest application scenarios are storyboarding and previsualization (35 tasks), product launches (21), game assets (21), e-commerce visuals (20) and real-estate marketing (19). Others include scientific illustration (18), renovation proposals (18), classroom teaching (16), and safety training, exhibition planning and construction reporting (15 each).^[1]

Each task has between 6 and 22 criteria, with a mean of 11.9. Function and geometry together make up 41.7% of all criteria, and 83.3% carry major importance (weight 3). By evidence route, 1,107 criteria (41.8%) are judged from rendered views and 1,544 (58.2%) by scene inspection. Requirements taken from specification files account for 500 criteria (18.9%).^[1]

Evaluation protocol

Generation and evaluation run in separate containers, and agents never see the rubrics during generation.^[1] Scoring has three stages:

• Automatic Agent Render (AaR). A rendering agent reads the task, rubrics, references and submission, picks eight complementary views, and renders each at 1,024 by 1,024 pixels with 64 samples in Cycles. It may move cameras and change render settings but must not alter the submitted geometry, materials, lighting, animation or color management. Each view can be re-rendered at most twice, and defects in the submission must not be repaired. If no new renders are produced, the evaluation counts as failed.^[1][3]
• Evidence-routed rubric judging. An Agent-as-Judge handles scene_summary criteria by inspecting the Blender project with tools and recording evidence about its internal state. A VLM-as-Judge handles render criteria using the AaR views, the agent's own exported media, and frames sampled from exported videos. Where visual evidence conflicts, the agent's original exported media take precedence. Unsupported rubric claims score zero.^[1]
• Dimension judging. The vision judge also rates eight dimensions from 1 to 10: scene fidelity, physical plausibility, style and color, realism, spatial layout, proportion and scale, lighting and materials, and completeness.^[1]

The task score is the equal-weighted mean of a rubric score and a dimension score, both on a 0-100 scale:^[1][2]

Rubric score    = 100 x sum(weight x pass) / sum(weight)
Dimension score = 10 x mean(eight ratings from 1 to 10)
Final score     = 0.5 x rubric score + 0.5 x dimension score

The benchmark score is the mean over all 223 tasks. Failed evaluations and submissions without a .blend file score zero and stay in the denominator.^[1][3] The paper used GPT-5.6-terra, a model in OpenAI's GPT-5.6 family, for all three evaluator roles, and the public evaluator defaults to the same model.^[1][3]

Judge agreement with humans

To validate the judge, the authors collected 150 pairs of outputs for the same input, each rated by three human reviewers. Three pairs with no majority were dropped, leaving 147.^[1]

The authors note that agreement falls when two outputs have similar average scores, and that dimension scores agreed with humans more often than rubric scores for every judge.^[1]

Results

All agents ran on the same 223 tasks, once per task, at their highest available reasoning setting. GPT-6-Astra and GPT-5.6-Sol used OpenAI's Codex CLI; the other eleven used Claude Code. Each agent worked in an isolated Linux container with one NVIDIA GPU and Blender 4.2.10 LTS, with shell, file, image-viewing and Blender MCP tools.^[1] The scores below are the authors' own results, using the model names as the paper labels them.^[1]

"Text" covers the 78 tasks without image or video references and "Multimodal" the other 145. "Full pass" is the share of tasks on which every rubric criterion passed; it does not mean an overall score of 100.^[1][4]

The authors' main observations:^[1]

• GPT-6-Astra's lead over Opus-5 is larger on multimodal tasks (80.7 against 72.1) than on text-only tasks (78.6 against 74.8). It ranks first on both the rubric and dimension components.
• GPT-5.6-Sol and DeepSeek-V4.1-Flash tie on multimodal tasks, but GPT-5.6-Sol averaged 39.9K output tokens and 44.0 tool calls per task against 209K tokens and 168 calls. The authors suggest the GPT models may plan more before acting, while DeepSeek-V4.1-Flash relies more on repeated execution and inspection.
• GPT-6-Astra fully passed 94 tasks and missed exactly one criterion on 64 more; Opus-5 had 57 and 51. Of the full passes, 40 were shared by both models, 54 were GPT-6-Astra only and 17 Opus-5 only.
• Realism was the lowest-rated quality dimension for every system; GPT-6-Astra averaged 6.61 out of 10 on realism and 7.89 on completeness.
• Reweighting rubric against dimension (weights from 0 to 1) leaves the ranking unchanged except for the order of Grok-4.6 and Opus-4.8. Across 2,899 individual outputs, the Spearman correlation between rubric and dimension scores was 0.630.

By category

GPT-6-Astra led eight of the nine categories. Its scores ranged from 71.1 in Rendering & Simulation to 86.8 in Procedural & Game Assets. Rendering & Simulation was the one category where Opus-5 scored higher (74.5 against 71.1).^[1] Every system scored higher on Procedural & Game Assets than on Art & Animation or Characters & Animals.^[1] Qwen3.8-Max scored above DeepSeek-V4.1-Flash on Nature & Landscapes despite ranking below it overall.^[1] Averaging the nine categories with equal weight changes only one position: Qwen3.8-Max and GLM-5.3-Flash swap fifth and sixth place.^[1]

Tool use

GPT-6-Astra and Opus-5 spent similar shares of their tool calls on Blender execution and image viewing, but Opus-5 made roughly twice as many executions (45.5 against 23.3 per task) and image views (27.1 against 15.0). Grok-4.6 had the largest share of asset operations (13%).^[1] Seed2.1pro-0915 and DeepSeek-V4.1-Flash made the most visual queries (33.9 and 31.3 per task) without high multimodal scores. The authors say this shows more visual inspection does not by itself coincide with higher scores, though aggregate counts cannot show how well an agent uses the feedback.^[1]

Failure modes

A qualitative audit found four recurring failures:^[1]

The appendix adds a case of "specification drift". A brief asked for 150 mm wooden legs plus brass casters. Opus-5's task plan instead specified 150 mm including the casters, built 102 mm of timber on 48 mm casters, and its self-check passed the 150.0 mm total. The frozen scene inspection measured the timber and failed the criterion.^[1] The authors conclude that current agents "tend to rely on 2D visual shortcuts to make things look right at a glance, but they still struggle to build genuinely solid and physically sound 3D scenes."^[1]

Limitations and future work

The paper's future-work section names five directions:^[1]

• Interactive and parametric evaluation. Giving the judge computer-use tools to manipulate rigs, animations and simulations, adding deterministic scripts for dimensions and tolerances, and adding reference solutions and geometric metrics such as voxel intersection-over-union for precision tasks.
• More complex and varied tasks. Localized editing, mesh repair, topology optimization, retopology, engineering drawings with tolerances, and revisions to previously delivered projects.
• Expert-aligned rubrics. Refining rubrics with professional 3D artists and exploring graded assessment where binary checks discriminate poorly, such as organic subjects.
• Other 3D applications. Supporting Maya, 3ds Max, more Blender versions and MCP tools, and external asset libraries such as Mixamo.
• Broader coverage. More models, reasoning-effort settings and harnesses; repeated runs to estimate variance and ranking stability (each agent was run once per task in the paper); Windows and macOS; and graphical-interface operation of Blender.

Running the benchmark

The repository provides a bw command-line tool to download and verify the dataset, prepare a task workspace, package a submission and run the evaluator. Each submission is a ZIP with final.blend at its root plus optional textures and outputs, and any generation harness can be used.^[2] Evaluation requires Linux, Python 3.10 or later, Blender 4.2.10, Node.js 22.17.0, ffmpeg and Claude Code 2.1.220, which drives the rendering and scene-inspection agents, plus a vision-capable chat-completions endpoint for the visual judge. The evaluator checks these versions before any model call and aborts on a mismatch.^[2][3] An optional integration with the Harbor evaluation framework is published on Hugging Face. It requests one GPU each for the generation and verifier environments, with separate 8-hour timeouts per task for generation and verification.^[5]

Related benchmarks

The paper's comparison table lists these programmatic 3D benchmarks:^[1]

BlenderGym, by Gu, Huang, Je, Yang and Guibas, was posted to arXiv in April 2025 as "BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing".^[8] The paper also cites the Blender agents SceneCraft and BlenderAlchemy, and productivity benchmarks such as WebArena, WorkArena and OSWorld.^[1] For the research tradition of recovering editable scene descriptions from images, see inverse graphics.

Relation to Qwen models

Two Qwen models are among the 13 evaluated agents. In the authors' results, Qwen3.8-Max ranked fifth with 60.6, and Qwen3.8-27B ranked tenth with 47.7.^[1] The paper's introduction cites the Qwen team's Qwen3.8-Max release post, together with OpenAI's GPT-6 Astra announcement, as examples of visual agents that interpret references, operate design tools and refine their outputs through visual inspection.^[1]

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46Fang, X., Wang, Y., Yu, W., et al. "BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References". arXiv:2610.23986, submitted 14 October 2026. Full text: arxiv.org/...2610.23986v1
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8BlenderWorld repository and README. QwenLM, GitHub. Accessed 25 October 2026.
3. ^1 ^2 ^3 ^4"Evaluation protocol and runtime". BlenderWorld documentation, GitHub. Accessed 25 October 2026.
4. ^1 ^2BlenderWorld project website and leaderboard. Accessed 25 October 2026.
5. ^1 ^2 ^3Qwen/BlenderWorld dataset card. Hugging Face. Accessed 25 October 2026.
6. ^Qwen Developers (@QwenDevs) post on X. 23 October 2026.
7. ^1 ^2"223道题、2651条标准：阿里Qwen给AI出了一套3D建模大考". IT Home, republished on ifeng.com, 17 October 2026.
8. ^Gu, Y., Huang, I., Je, J., Yang, G., Guibas, L. "BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing". arXiv:2504.01786, 2025.