On September 4, the atopile team released EEBench V1: 13 hardware design tasks where AI models write circuits as code, and SPICE simulation—not another language model—grades the results. The motivation: GPT-6 Astra's launch demo showed PCB editing in KiCad, and rather than argue about whether that's real, atopile decided to test it.
atopile is a small YC W24 team behind an open-source language for "drawing circuit boards with code" (3,700+ GitHub stars).
How the Exam Works
The pipeline is rigorous: the model submits atopile code, which is compiled into a schematic and BOM, then simulated in ngspice under both nominal conditions and worst-case tolerance corners. Voltage limit violations, insufficient capacitance, and cost overruns are all decided by the machine. Final score = technical score × 0.65 + cost score × 0.35, with cost benchmarked against reference designs at 100-unit distributor pricing. No human grading, no LLM judge.
The 13 problems were newly written in 2026 and never published, so models can't memorize them. Two revealed tasks: a hold-up capacitor array for a power meter (easy tier), and a multiple-feedback low-pass filter around an op-amp where every tolerance corner must stay within limits (hard tier).
The Leaderboard
| Model | Score (/100) | |---|---| | GPT-6 Astra | 69.3 | | Claude Opus 5 | 61.6 | | Grok 4.6 | 57.1 | | Claude Fable 5.1 | 56.4 | | Gemini 3.8 F | 55.4 | | GPT-5.5 | 42.3 | | GPT-5.6 Sol | 39.4 | | GLM 5.2 Max | 16.7 | | DeepSeek V4 | 10.5 |
Notable observations:
- Blink-and-you-miss-it suspense. The blog post published September 4 said they didn't yet have GPT-6 Astra's score—the same day, a regenerated live leaderboard put Astra on top at 69.3%, announced by atopile themselves in the HN comments. The blog's cliffhanger lasted less than a day.
- The self-reporting gap persists. xAI's Grok 4.6 model card claims 60.0%; atopile measured 57.1%—a 2.9-point gap between vendor self-tests and third-party evaluation, even on hardware problems.
- Open-weight models lag badly. GLM 5.2 Max scored 16.7 and DeepSeek V4 Pro 10.5. Models that nearly match closed frontier models on software benchmarks remain an order of magnitude behind in the analog world.
- "Anyone with a hobbyist's knowledge of component names knows this stuff."
- Models resemble someone who "memorized undergraduate and graduate textbooks but has zero hands-on experience and can't answer whether a design is good enough"—they still love the 741 op-amp.
- Others noted the benchmark doubles as atopile advertising: problems are written in their language, graded by their simulator, and the leaderboard drives consulting business. atopile's defense is verifiable: the blog states they pay for public runs and don't sell leaderboard placements. Both things can be true.
The Capacitor That Flunked
The benchmark report's most instructive failure: one model chose a nominal 22 µF ceramic capacitor for the hold-up task. The code compiled fine—but under a 4.7 V DC bias, the effective capacitance was only 11.4 µF, against a task requirement of 545 µF. The supply rail fell below the 3 V red line after 0.85 ms; the spec required 20 ms.
The math checks out: with a 93 Ω load discharging from 4.55 V to 3.0 V, RC decay gives 93 × 22µ × ln(4.55/3.0) = 0.85 ms. Exactly as measured. The judge didn't wrong it.
The root cause is a lesson for anyone doing hardware: high-value ceramic capacitors use barium titanate ferroelectric dielectrics, and DC bias pre-polarizes the domains, shrinking capacitance under voltage. Losing 40–60% at half rated voltage is normal for small X5R packages. The model could recite the 22 µF on page one of the datasheet, but not the bias derating curves on page ten. The missing course is called experience.
Community Reaction
The HN thread (384 points, 209 comments) featured sober practitioner takes:
Back to the title question. The blog's own closing says it best: "Some of them, yes. We still would not ask it to design a pacemaker and blindly install the result."
References: EEBench blog | Methodology | HN discussion | atopile