# Build-A-Chip-Bench

> Source: https://aiwiki.ai/wiki/build_a_chip_bench
> Updated: 2026-10-27
> Fact-checked: 2026-10-27
> Categories: AI Agents, AI Benchmarks, AI Hardware, Model Evaluation
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Build-A-Chip-Bench." aiwiki.ai, 27 Oct 2026. https://aiwiki.ai/wiki/build_a_chip_bench
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Build-A-Chip-Bench** is a benchmark for [AI agents](https://aiwiki.ai/wiki/ai_agents) that tests whether an agent can carry a digital chip design from a workload specification all the way to a routed, verified GDS layout, the layout data that represents the physical design.[4] It was developed by the semiconductor-design startup [ChipAgents](https://aiwiki.ai/wiki/chipagents), with [NVIDIA](https://aiwiki.ai/wiki/nvidia) contributing technical expertise to the benchmark suite and scoring rubric, and was announced on October 14, 2026.[1][2] Unlike earlier hardware benchmarks that grade only register-transfer-level (RTL) code generation or verification, Build-A-Chip-Bench requires a design to pass functional verification and the physical-design flow before it is scored on power, performance and area (PPA). In ChipAgents' initial results, the best agent, built on [Claude Fable 5.1](https://aiwiki.ai/wiki/claude_fable_5_1), reached an overall score of 16.6%.[2][3]

| | |
|---|---|
| Developer | ChipAgents, with technical expertise from NVIDIA[1] |
| Authors | Surya Gunukula, Jerry Yicheng Wang, Kevin Zhang (Gunukula and Wang credited as equal contributors)[2] |
| Announced | October 14, 2026[1] |
| Publication | ChipAgents Research blog post, October 2026[2] |
| Scope | Workload specification to GDS layout, 12 design-flow steps[2] |
| Tools | Open-source tooling and an open-source process design kit (PDK)[2] |
| Agent harness | [OpenCode](https://aiwiki.ai/wiki/opencode) for all models tested[2] |
| Top initial score | 16.6% (Claude Fable 5.1), self-reported by ChipAgents[2] |

## Background

Earlier benchmarks for language models in hardware design concentrate on writing or checking Verilog. NVIDIA researchers released VerilogEval in September 2023, a set of 156 problems from the HDLBits instructional site that checks generated Verilog against a golden solution in simulation.[5] RTLLM, released the same year, is an open-source benchmark for generating design RTL from natural-language instructions.[6] NVIDIA's Comprehensive Verilog Design Problems (CVDP), published in June 2025, expanded the scope to 783 problems across 13 task categories covering RTL generation, verification, debugging, specification alignment and technical Q&A, in both non-agentic and agentic formats.[7] ChipBench, posted to arXiv in January 2026, evaluates Verilog generation (44 hierarchical modules), debugging (89 cases) and reference-model generation (132 samples), and reported that Claude Opus 4.5 reached 30.74% on its Verilog generation task.[8]

ChipAgents argues that VerilogEval, CVDP and ChipBench are "point tests" that cover only the RTL and verification steps and stop before physical implementation.[2] Its post says recent models have plateaued at around 65-70% pass@3 on agentic CVDP, that many of the remaining CVDP tasks appear to have problems such as specification ambiguity or malformed testbenches, and that ChipBench scores rose from 30.7% at release to 90.6%, leaving little headroom.[2] Those trend figures are ChipAgents' own compilation. The company's press release states the gap plainly: RTL that simulates correctly "can still miss timing, exceed its area or power budget, fail to route, or produce physical-design violations."[1]

For other AI approaches to chip design, see [ChipNeMo](https://aiwiki.ai/wiki/chipnemo), NVIDIA's family of domain-adapted language models for chip-design tasks, and [AlphaChip](https://aiwiki.ai/wiki/alphachip), Google DeepMind's reinforcement-learning method for macro placement.

## Task design

Each task starts from a workload specification and ends with a GDS layout, and the agent owns every step in between. ChipAgents lists twelve steps: workload specification, architecture, microarchitecture, RTL design, functional verification, logic synthesis, floorplanning, placement, clock tree synthesis, routing, timing and power signoff, and physical verification.[2] The post stresses that problems found late often trace back to earlier choices, for example timing failures at signoff caused by a shallow pipeline, or routing congestion caused by the floorplan, so the agent must sometimes go back and revise earlier steps.[2]

Each task runs across three isolated containers:[2]

| Container | Contents | Agent access |
|---|---|---|
| Agent container (sandboxed) | Workload specification, pinned interface the design must attach to, open-source tooling (simulation, synthesis, place and route, signoff) and an open-source PDK | Full |
| Verification container | Hidden test vectors and a hidden verification IP (VIP) | Can send requests, cannot inspect |
| Grader container | Grader and a reference design for each task, against which task constraints are calibrated so that they are known to be achievable | None |

Tasks have a fixed time budget, and the sample runs shown in the post are 72-hour runs. ChipAgents notes that synthesis and place-and-route consume most of the budget, with a single run taking hours, so an agent cannot simply sweep settings until timing closes.[2]

### Example task: a Qwen3 inference accelerator

The post publishes a heavily summarized task card in which the agent must build an inference accelerator [ASIC](https://aiwiki.ai/wiki/asic) for a pinned [Qwen3](https://aiwiki.ai/wiki/qwen3)-0.6B INT4 checkpoint, from the checkpoint to a routed GDS. Only the top-level interface is fixed; the agent chooses array shape, tiling, buffering, scheduling, numerics and weight layout. The run is network-isolated and the agent gets Verilator, OpenROAD and the Nangate45 platform, but no held-out tests, expected outputs, numerical oracle, physical design or reference RTL. The objective is to maximize decode tokens per second once all constraints are met.[2]

| Constraint | Target | Sample reference design |
|---|---|---|
| Process platform | Nangate45, OpenROAD flow | Same |
| Clock | 400 MHz, setup WNS at least 0 ns, no TNS | +0.131 ns setup WNS |
| Die area | At most 6 mm² | 5.76 mm² |
| Power | Under 10 W | 136.2 mW |
| Decode throughput | At least 15 tokens/s | 16.35 tokens/s |
| Physical checks | Zero DRC errors, zero unrouted nets | 0 DRC errors, 0 unrouted nets |
| Held-out functional tests | Must pass | 56 of 56 prompts matched |

The reference design, which has 12 modules, 1,796 lines of SystemVerilog and 25 SRAM macros, was created by Jerry Yicheng Wang, described as Lead Digital Designer at ChipAgents, with AI assistance.[2] Deliverables include synthesizable RTL, a software oracle and self-written testbenches, deliberate mutants that the tests must catch, the OpenROAD flow configuration, the final GDS, DEF, netlist and SPEF files, and architecture, verification and machine-readable PPA reports.[2]

## Scoring

Scoring has two stages. A design must first pass a hard gate made of two checks: functional verification against the hidden VIP, and a "GDS pass", meaning the design completes the physical flow to a GDS layout that routes, has no hold-time violations and is free of design-rule (DRC) violations. Designs that pass both are then scored on PPA against the task's constraints, such as throughput, clock target, power and area.[2]

The overall score is the mean over all N tasks of f × g × p, where f is 1 if the design passes functional verification, g is 1 if it produces a clean GDS, and p is the PPA-versus-constraint score from 0% to 100%. A task that fails either gate scores zero, so strong PPA on a few tasks cannot make up for failures elsewhere.[2]

## Initial results

ChipAgents ran each model with OpenCode as the agent harness and published the following results. They are self-reported by the benchmark's developer, and ChipAgents said more model results were "coming soon."[2]

| Model | Functional pass | GDS pass | PPA vs. constraint | Score |
|---|---|---|---|---|
| [Claude Fable 5.1](https://aiwiki.ai/wiki/claude_fable_5_1) | 65% | 75% | 25.5% | 16.6% |
| [GPT-6 Astra](https://aiwiki.ai/wiki/gpt_6_astra) | 85% | 85% | 10.2% | 8.7% |
| [Gemini 3.8 Flash](https://aiwiki.ai/wiki/gemini_3_8_flash) | 20% | 45% | 4.1% | 0.8% |
| [Kimi K3](https://aiwiki.ai/wiki/kimi_k3) | 25% | 25% | 1.3% | 0.2% |
| [GLM-5.3](https://aiwiki.ai/wiki/glm_5_3) | 15% | 5% | 0.0% | 0.0% |

Functional pass and GDS pass are shares of all tasks. PPA versus constraint is averaged only over designs that pass both gates, which ChipAgents notes is "not meaningful on its own."[2] GPT-6 Astra passed both gates more often than Claude Fable 5.1 but scored lower overall because its passing designs achieved much weaker PPA. The post's abstract says the best agent passes the hard gate on 65% of tasks.[2]

ChipAgents' press release summarizes the same evaluation by saying tested agents passed functional verification on about 40% of tasks on average, while about 10% met the required PPA targets.[1] The Information reported that the Fable 5.1-powered agent was the best performer at 16.6%.[3]

The post also plots each model's best self-measured score over the first 36 hours of 72-hour runs on two sample tasks. On those tasks Claude Fable 5.1 reached 61 and 67, GPT-6 Astra 47 and 32, Gemini 3.8 Flash 14 and 6, and GLM-5.3 reached 13 on the second task; a model absent from a panel never produced a passing design on that task. These curves use the agents' own measurements before hidden-VIP grading.[2]

## Reported failure modes

ChipAgents described four patterns from its runs, each illustrated with an agent trace:[2]

| Finding | What ChipAgents observed |
|---|---|
| Closed frontier models can complete the full flow | Closed frontier models can generally take a design from specification through RTL, synthesis and physical design to GDS, and backend tool use is "reasonably reliable." They sometimes choose sensibly between rewriting RTL and applying a targeted engineering change order (ECO), though not consistently. Open models struggle with the frontend and more with the backend. A GPT-6 Astra trace shows Yosys synthesis followed by the OpenROAD physical flow and a post-route ECO that cleared a remaining slew violation. |
| Chasing PPA breaks correctness | Agents rarely treat keeping existing behavior intact as a constraint while optimizing. In one run (the trace shown is from Kimi K3), the agent rewrote its datapath aiming for about double the throughput, introduced a functional bug and a hang, spent over an hour debugging, and finished about 3% slower than where it started. |
| Physical failures traced back to RTL too late | Agents respond to timing, area or routing failures by re-running place-and-route with new settings first. In one run (trace from GPT-6 Astra), the agent documented a storage change that would cut SRAM macros from 51 to 27, deferred it, and spent about six hours on routing configurations while detailed routing reported hundreds of thousands of violations before making the change. |
| Slow tool loops break convergence | Agents bring fast try-and-retry habits to a flow where one place-and-route run can take more than four hours. They underestimate slow steps, start runs they cannot finish and abandon optimizations partway. A Claude Fable 5.1 trace shows the agent killing a clock-tree-synthesis run after hold repair left the worst slack unchanged at -0.987 ns between iterations 810 and 1,750. |

## Statements and reception

Announcing the benchmark, William Wang, founder and CEO of ChipAgents, said "the real challenge is carrying engineering intent through a long sequence of interconnected decisions until the design satisfies both functional and physical requirements." Tim Costa, vice president and general manager of industrial and computational engineering at NVIDIA, said the benchmark "tests whether agents' decisions hold up from design intent through physical implementation."[1] ChipAgents frames the benchmark as a way to measure progress toward what it calls "chip superintelligence."[1]

In an interview with The Information, Wang said AI had already cut chip development timelines from at least two years to about one year. He said he expects AI advances to compress the process to four to six months in the coming year.[3] These are Wang's predictions, not results from the benchmark.

Writing in SemiWiki, Daniel Nenni noted that the 40% and 10% figures "are company-reported results under the benchmark's conditions" and "should not be interpreted as universal success rates for AI chip design." He added that comparisons would be more informative if they also disclosed compute budgets, tool configurations and repeatability.[4]

## Limitations

- **Self-reported, single-developer results.** All published scores come from ChipAgents, which also sells an agentic chip-design platform.[1][2]
- **Limited disclosure.** As of October 27, 2026, the benchmark is documented in a company blog post and press release. The post does not state the total number of tasks, and its example task card is published only in heavily summarized form.[2]
- **Fixed harness and open-source flow.** Every model was run inside the same OpenCode harness, so scores reflect a model plus that harness. Tasks use open-source tools and an open-source PDK (Nangate45 in the example) rather than commercial EDA tools.[2]
- **Small model set.** Five models were reported at launch.[2]

## Future plans

ChipAgents says it is working with industry partners to launch a live leaderboard covering more frontier models and an expanded task set.[2] The post's acknowledgements list Analog Devices, MediaTek and NVIDIA as industry collaborators that provided technical advice and support.[2]

## About ChipAgents

ChipAgents, founded in 2024, sells an agentic AI platform for semiconductor design and verification.[1] In July 2026 it said it had expanded its Series A to $134 million with a $60 million Series A2 that brought in new investor B Capital, and that it had deployments at more than 120 semiconductor companies.[9] See [ChipAgents](https://aiwiki.ai/wiki/chipagents) for details.

## See also

- [ChipNeMo](https://aiwiki.ai/wiki/chipnemo)
- [AlphaChip](https://aiwiki.ai/wiki/alphachip)
- [Ricursive Intelligence](https://aiwiki.ai/wiki/ricursive_intelligence)
- [AI benchmarks](https://aiwiki.ai/wiki/benchmark)
- [SWE-bench](https://aiwiki.ai/wiki/swe_bench)
- [Terminal-Bench](https://aiwiki.ai/wiki/terminal_bench)

## References

1. ChipAgents, "ChipAgents Collaborates With NVIDIA to Launch Build-A-Chip-Bench for Autonomous Chip Design and Implementation," press release, October 14, 2026. https://chipagents.ai/newsroom/chipagents-nvidia-build-a-chip-bench
2. Surya Gunukula, Jerry Yicheng Wang and Kevin Zhang, "Introducing Build-A-Chip-Bench: from workload specification to GDS," ChipAgents Research, October 2026. https://chipagents.ai/research/build-a-chip-bench
3. Phoebe Liu, "Exclusive: Nvidia and ChipAgents Launch Evaluation For AI-Powered Chip Design," The Information, October 14, 2026. https://www.theinformation.com/briefings/exclusive-nvidia-chipagents-launch-evaluation-ai-powered-chip-design
4. Daniel Nenni, "ChipAgents and NVIDIA Put Autonomous Chip Design to the Test," SemiWiki, October 2026. https://semiwiki.com/eda/chipagents-ai/374909-chipagents-and-nvidia-put-autonomous-chip-design-to-the-test/
5. Mingjie Liu, Nathaniel Pinckney, Brucek Khailany and Haoxing Ren, "VerilogEval: Evaluating Large Language Models for Verilog Code Generation," arXiv:2309.07544, September 2023. https://arxiv.org/abs/2309.07544
6. Yao Lu, Shang Liu, Qijun Zhang and Zhiyao Xie, "RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model," arXiv:2308.05345, August 2023. https://arxiv.org/abs/2308.05345
7. Nathaniel Pinckney et al., "Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification," arXiv:2506.14074, June 2025. https://arxiv.org/abs/2506.14074
8. Zhongkai Yu et al., "ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design," arXiv:2601.21448, January 2026. https://arxiv.org/abs/2601.21448
9. ChipAgents, "ChipAgents Expands Series A Funding to $134 Million as Demand Grows for Agentic AI in Semiconductor Design," July 29, 2026. https://chipagents.ai/newsroom/chipagents-series-a2-134m