Can AI Design Circuit Boards Yet? EEBench V1 Uses SPICE Simulation as the Judge

Forum topic · QianXun · 2026-09-06

Summary

On September 4, the atopile team (YC W24, makers of a code-based circuit board language) released EEBench V1, a benchmark of 13 newly written, never-published hardware design tasks. AI models submit circuits as atopile code, which is compiled into schematics and BOMs, then judged by ngspice simulation under both nominal and worst-case tolerance conditions — no human or LLM graders. Scores combine technical performance (65%) and cost (35%) against distributor pricing at 100-unit quantities. On the live leaderboard, GPT-6 Astra leads at 69.3%, followed by Claude Opus 5 (61.6) and Grok 4.6 (57.1, versus xAI's self-reported 60.0). Open-weight models score far lower: GLM 5.2 Max at 16.7 and DeepSeek V4 Pro at 10.5. The report's standout failure: a model chose a 22 µF ceramic capacitor for a hold-up task, but DC bias derating reduced its effective capacitance to 11.4 µF against a 545 µF requirement, causing the power rail to crash below 3 V in 0.85 ms instead of the required 20 ms. Hacker News commenters praised the physics-based grading while noting the benchmark doubles as atopile self-promotion.

The benchmark: physics as the judge

After AI saturated software benchmarks like SWE-bench, the atopile team — a YC W24 startup behind an open-source "circuits as code" language with 3,700+ GitHub stars — moved the exam into hardware. Published September 4, EEBench V1 consists of 13 hardware design tasks written fresh in 2026 and never released publicly, so models cannot memorize them. The motivation: GPT-6 Astra's launch demo showed PCB editing in KiCad, and rather than argue about whether that's real, they decided to test it.

The grading pipeline is unforgiving:

1. The model submits atopile code. 2. Code compiles into a schematic and BOM. 3. ngspice simulates the circuit — once at nominal operating conditions, once at worst-case tolerance corners. 4. Voltage limits, capacitance adequacy, and cost are all checked mechanically. 5. Final score = 0.65 × technical score + 0.35 × cost score, with cost based on 100-piece distributor pricing compared to a reference design.

No human grading, no LLM judges — only physical law. Only two of the 13 task names are public: a hold-up capacitor array for a power meter (easy tier) and a multiple-feedback low-pass op-amp filter where every tolerance corner must stay within limits (hard tier).

Leaderboard

| Model | Score (/100) | |---|---| | GPT-6 Astra | 69.3 | | Claude Opus 5 | 61.6 | | Grok 4.6 | 57.1 | | Claude Fable 5.1 | 56.4 | | Gemini 3.8 F | 55.4 | | GPT-5.5 | 42.3 | | GPT-5.6 Sol | 39.4 | | GLM 5.2 Max | 16.7 | | DeepSeek V4 | 10.5 |

Notable observations:

Suspense with a one-day lifespan. The blog post originally said GPT-6 Astra had no score yet; the same day, the regenerated live leaderboard showed Astra on top at 69.3% — announced by atopile staff in the HN thread.
Self-reported vs. third-party gap. xAI's Grok 4.6 model card claims 60.0%; atopile measured 57.1% — a 2.9-point gap showing vendor self-testing and independent evaluation still diverge on hardware tasks.
Open-weight models lag by an order of magnitude. Models approaching closed-source performance on software code collapse to double-digit-low or single-digit scores here.

The capacitor that failed in front of the judge

The benchmark report's most instructive failure: a model picked a nominal 22 µF ceramic capacitor for the hold-up task. The code compiled fine. The simulation showed that under a 4.7 V DC bias, effective capacitance was only 11.4 µF — against a 545 µF requirement. The power rail fell below the 3 V red line after 0.85 ms; the spec demanded 20 ms.

Independent check: with a 93 Ω load decaying from 4.55 V to 3.0 V, the RC discharge formula gives 93 × 22 µF × ln(4.55/3.0) ≈ 0.85 ms. The judge was exactly right.

The root cause is a lesson for anyone doing hardware: high-capacitance ceramics use barium titanate ferroelectric dielectrics, and DC bias pre-polarizes the domains, shrinking capacitance — small-package X5R parts routinely lose 40–60% at half rated voltage per manufacturer derating curves. The model could recite page one of the datasheet (22 µF) but not page fifteen's bias derating curve. The missing subject is called experience.

Community reaction

The HN thread (384 points, 209 comments) brought practitioner skepticism:

"Any hobbyist who knows the component name understands this."
Models resemble someone who "memorized undergrad and grad textbooks but has zero hands-on experience and can't judge whether something is good enough" — they still love the 741 op-amp.
Conflict-of-interest concerns: the tasks are written in atopile's language, judged by their simulator, and the leaderboard drives consulting business. atopile's verifiable rebuttal: the blog states "we pay for public benchmark runs and do not sell leaderboard scores." Both things can be true.
A version warning hidden in the changelog: on the July pre-release board, Claude Fable 5 scored 71.7%; on the September V1 task set, the same model scored 54.3%. Scores are not comparable across versions — check the version number before reading any leaderboard.

So, can AI design circuit boards?

The blog's own closing says it best: "Some of them, yes. We still would not ask it to design a pacemaker and blindly install the result."

References: EEBench blog | Methodology | HN discussion | atopile