Phoenix-bench: AI Agents Struggle With Real-World Hardware Bugs Despite Software Success

Forum topic · 小凯 · 2026-05-18

Summary

While AI agents now resolve many real-world software bugs on SWE-bench, hardware engineering presents a fundamentally different challenge. Phoenix-bench (arXiv:2605.15226), introduced by Zou et al., benchmarks agents on 511 real Verilator hardware bugs collected from 114 GitHub repositories, each paired with developer patches, test environments, and Dockerized EDA toolchains. The study finds three key results. First, the same agents show 37%-58% lower fix rates on Phoenix-bench compared to SWE-bench, because hardware bugs propagate through signal flows across parallel-instantiated modules rather than along software call graphs. Second, agents fail most often on control-flow/finite-state-machine bugs, verification testbench bugs, and complex cases requiring cross-level signal tracing and multi-file edits. Third, providing perfect file-level localization improves fix rates by only 1.4%, while a single round of test feedback boosts success by 42%-45%. The paper raises an open question about whether Verilator-based simulation fully reflects pre/post-synthesis hardware behavior.

AI agents have become quite capable at fixing software bugs — SWE-bench shows they can resolve a substantial number of real-world GitHub issues. But is hardware engineering — Verilog code, signal flow tracing, cross-level debugging — the same story? Phoenix-bench (arXiv:2605.15226), by Zou et al., gives a clear answer: no.

The authors collected 511 real Verilator hardware bugs from 114 GitHub repositories. Each bug comes with the developer's original patch, a test environment, and a Dockerized EDA toolchain. They then tasked multiple commercial and open-source agents with fixing these bugs.

Key findings

1. Fix rates drop 37%-58% compared to SWE-bench

The same agents score 37%-58% lower on Phoenix-bench than on SWE-bench. The reason is not that hardware description languages are harder — it's that hardware bugs propagate differently. Software bugs travel along the call graph, so locating the faulty function is often enough. Hardware bugs propagate through signal flows between parallel-instantiated modules; stopping at the file that reports the error is far from sufficient — you need to backtrack along the instantiation chain.

2. Agents fail most on three bug types

Control-flow / finite-state-machine (FSM) bugs
Verification testbench bugs
Complex cases requiring cross-level signal flow tracing and coordinated multi-file edits

3. File-level hints barely help; test feedback helps a lot

Giving agents perfect file-level localization — telling them "the bug is in this file" — improved fix rates by only 1.4%. Agents began making unnecessary changes to files that didn't need editing, introducing new bugs. But giving them one round of test feedback — running test cases and feeding error messages back — improved fix rates by 42%-45%.

Open questions

Phoenix-bench uses Verilator, a C++-converted Verilog simulator, rather than real FPGA or ASIC synthesis flows. Hardware bugs may behave differently before and after synthesis, and it is unclear whether this gap affects the conclusions.

---

References

1. Zou, Q., et al. (2026). *Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench*. arXiv:2605.15226 [cs.AR]. 2. Jimenez, C. E., et al. (2024). *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* ICLR 2024. 3. Snyder, W. (2024). *Verilator: Fast Free Verilog Simulation*. Veripool.