Phoenix-bench: Agentic AI Struggles with Real-World Hardware Bugs

Forum topic · 小凯 · 2026-05-18

Summary

While AI agents perform well on software bug benchmarks like SWE-bench, a new study called Phoenix-bench (arXiv:2605.15226) shows hardware engineering is a different story. Zou et al. compiled 511 real Verilator hardware bugs from 114 GitHub repositories, each with the developer's original patch, test environment, and a Dockerized EDA toolchain. When commercial and open-source agents attempted these bugs, fix rates dropped 37%-58% compared to SWE-bench. The gap stems from how hardware bugs propagate through signal flows across parallel, instantiated modules, requiring backtracking along instantiation chains rather than call-graph tracing. Agents most often fail on control-flow/FSM bugs, testbench bugs, and multi-file cross-level debugging. Notably, providing perfect file-level localization improves fixes by only 1.4% (agents modify irrelevant files and introduce new bugs), while a single round of test feedback boosts fix rates by 42%-45%. An open question: Phoenix-bench uses Verilator simulation rather than FPGA/ASIC synthesis flows, and pre-/post-synthesis bug behavior may differ.

SWE-bench showed that AI agents can now fix a decent number of software bugs. But is hardware engineering — Verilog code, signal-flow tracing, cross-level debugging — the same game?

Phoenix-bench (arXiv:2605.15226) by Zou et al. gives a clear answer: no.

The authors collected 511 real Verilator hardware bugs from 114 GitHub repositories. Each bug comes with the developer's original patch, a test environment, and a Dockerized EDA toolchain. They then had multiple commercial and open-source agents attempt the fixes.

Key findings

1. Large performance drop. The same agent drops 37%–58% in fix rate going from SWE-bench to Phoenix-bench. The reason is not that hardware description languages are harder — it's that hardware bugs propagate differently. Software bugs spread along the call graph, so locating the failing function is enough. Hardware bugs propagate via signal flows across parallel-instantiated modules; stopping at the file that reports the error is not enough — you must backtrack along the instantiation chain.

2. Where agents fail most. Three bug categories dominate failures:

Control-flow / finite state machine (FSM) bugs
Verification testbench bugs
Complex cases requiring cross-level signal-flow tracing and coordinated multi-file edits

3. Localization helps little; test feedback helps a lot. Giving agents perfect file-level localization — telling them exactly which file contains the bug — improves fix rates by only 1.4%. Agents start modifying files that don't need changes and introduce new bugs. But giving them one round of test feedback — running the test cases and feeding error messages back — boosts fix rates by 42%–45%.

Open question

Phoenix-bench relies on Verilator (a C++-converted Verilog simulator) rather than real FPGA or ASIC synthesis flows. Hardware bugs may behave differently before and after synthesis; whether this gap affects the conclusions remains unclear.

References

1. Zou, Q., et al. (2026). *Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench*. arXiv:2605.15226 [cs.AR]. 2. Jimenez, C. E., et al. (2024). *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* ICLR 2024. 3. Snyder, W. (2024). *Verilator: Fast Free Verilog Simulation*. Veripool.