The Final Court of Physics: 91% Submitted, 21% Passed
> Part three of the trilogy "When the World Stops Being Forgiving": in the final court, AI discovers that the laws of physics won't even grant it an appeal.
⚖️ Exams vs. Acceptance Testing
First, a truth every engineer understands but many AI researchers dislike hearing: passing an exam and passing acceptance testing are two different things.
An exam has a question-setter with a standard answer in mind. Acceptance testing has no question-setter—or rather, the setter is the laws of physics themselves, which never hand out question papers, only verdicts. If your chip comes back from the fab not working, there is no "partial credit." If your layout shorts in the wafer fab, there is no "credit for the idea." Physics is not an examiner; physics is the final judge: it doesn't explain, doesn't console, doesn't offer a retake, and speaks only through "works" or "doesn't work." In the software world, bugs are cheap—discover, locate, fix, hot-patch, a loop measured in hours or minutes. In the hardware world, bugs are expensive—a single tape-out costs tens of thousands to millions of dollars, cycles are measured in months, and the dead chip rests forever in the archive of failed projects.
This distinction is why AI's glorious record in software doesn't automatically convert into a pass for the hardware world. A model that writes beautiful LeetCode solutions may not know how many vias a 30mA current path needs; an agent that debugs large codebases may not understand a phase-margin warning from ngspice. Software errors throw exceptions; hardware errors fail silently.
This is exactly what makes analog IC layout design so brutal. Unlike digital circuits—whose layout is largely automatable because digital logic is discrete, regular, and built on reusable standard-cell libraries—analog layout remains highly dependent on human craftsmanship. Why? Because analog performance is continuous, sensitive, and governed by parasitics: place two differential-pair devices slightly apart and mismatch creeps in; route a critical signal near a power line and noise couples in; skimp on vias and current density exceeds limits, leading eventually to electromigration failure. More subtly, these constraints often conflict—compact area versus good isolation is an eternal feud, matching precision versus routing length another. The veteran's value lies not in knowing the rules but in knowing which rule should yield when rules fight. That judgment is written about in textbooks, but the true sense of proportion lives only in the veteran's hands.
Because of this craftsmanship dependence, analog layout has been EDA automation's "last fortress"—digital synthesis and auto place-and-route are mature commercial products, while analog layout tools have remained at the "assisted drawing" level. There's a half-joking industry saying: digital designers measure output in layouts, analog designers measure it in years. A senior analog layout engineer often needs weeks to months from reading a schematic to delivering a qualified layout.
So the question becomes: can coding agents—triumphant in software, skilled at writing code, debugging, and running tests—chew through this hardest bone? The question matters far beyond EDA for AI for Science: errors in software tasks can be exposed and fixed at runtime, while errors in hardware tasks are sealed into silicon, and once taped out there is no patch. The "run it first, fix later" habit agents developed in the software world comes at a price in the hardware world.
The paper is called AnalogLayoutWorld, from the Nova-Silicon team. It does one thing: builds a final court of analog layout design for AI, puts the strongest 13 coding agents in it, and sees what verdicts they receive. The headline numbers are sobering: 91.1% of attempts successfully produced a GDS layout file, but only 21.3% passed all acceptance criteria.
Ninety percent submit, twenty percent pass. The courtroom door is open, but the verdict is harsh. Hidden in the gap between 91.1% and 21.3% is the most underestimated chasm facing coding agents entering the physical world: generating a file that "looks like a layout" and generating a layout that "survives physical inspection" are difficulties of entirely different magnitudes. GDS is just a geometric description—anything that can write a script can emit GDS. But DRC's geometric rules, LVS's topological consistency, PEX's parasitic extraction, and post-layout simulation's performance specs—each stage filters out another batch of "looks about right" submissions.
🏛️ Blueprint of the Court: 40 Tasks and Four Judges
The design of the court itself deserves scrutiny, because its rigor determines the credibility of the verdict.
The exam: 40 expert-reviewed analog layout tasks, all using fixed transistor-level netlists from the SKY130 open-source PDK—covering five categories: amplifiers, bandgap references, power management, sampling circuits, and oscillators. SKY130 wasn't chosen at random: it is SkyWater's open-source 130nm planar process with a fully public PDK and no licensing barriers, meaning any researcher worldwide can reproduce this court under identical conditions—reproducibility is a benchmark's first virtue. Fixed netlists mean every agent faces the same questions and cannot game the test by "picking an easy one." Tasks come from public benchmarks like Analog Design Bench and LOADBench, plus the team's own and open-source designs. Forty tasks isn't many, but each was reviewed by domain experts—quality over quantity is the foundation of this exam's credibility.
The examination room: each agent works in an isolated Docker container (orchestrated by Harbor) containing the SKY130 PDK and four tools: gdsfactory (layout manipulation), KLayout (DRC and LVS), Magic (parasitic extraction, PEX), and ngspice (circuit simulation). Note a key design decision: no benchmark-specific typed tools or MCP interfaces are provided—agents use a generic code-execution tool and must decide themselves which tools to call, in what order, and how to script the parsing of outputs. Why? Because specialized interfaces are like an open-book exam where someone flips the pages for you; they can't measure real engineering judgment. With MCP tools, you've handed over an index to the standard answers—in real engineering, nobody wraps the tools into pretty functions for you; you must read engine errors yourself, write parsing scripts yourself, and judge the next step yourself. This "no assistance" design means the benchmark measures the lower bound of general agent capability, not the upper bound of some specialized system.
Time limit: 3-hour wall-clock budget. Submission is saving layout.gds to a specified path.
The judges: four, all independent; agents' self-reported results count for nothing—this is the paper's hardest methodological principle. Why not trust self-reports? Because LLM hallucination in engineering contexts manifests as: misreading engine output (seeing "1000 errors" as "0 errors"), declaring "all passed" without running verification, or running the wrong rule file without knowing. Without independent judges, the grade sheet is graded by the examinee. The DRC judge checks geometric rules—line widths, spacings, via sizes. The LVS judge checks whether layout and schematic are topologically consistent—whether you drew what you think you drew. The PEX + post-simulation judge extracts parasitics and runs simulation—do gain, bandwidth, offset, phase margin meet spec? The geometric-quality judge measures two things: effective device area ratio ρ_t = A_eff/A_layout (how much of the layout is actually devoted to devices; higher is more compact) and aspect ratio (must not exceed 4—overly elongated layouts are painful to package). There is also a morphological screen targeting degenerate "bus-row" layouts, which arrange devices in a row connected by buses—looks tidy, wastes matching and compaction opportunities; discovery means immediate disqualification.
The scoring formula S_t aggregates the four judges' weighted opinions plus engineering-quality bonuses: device matching (measured by consistency of effective areas between devices), quality of critical-net routing, and power-delivery quality (via count versus current demand, with coupling penalties). The result is a total score from 0 to 1. This formula deserves a second look, because it separates "works" from "works well"—passing all acceptance is the threshold (eligibility); the total score is the ranking (merit). A benchmark testing only "does it run" teaches agents to submit sloppy work; the quality score forces them to think about trade-offs like real designers.
📊 The Verdict: A Score Sheet with a Glaring Gap
13 model-framework configurations, 3 trials per task—120 total attempts, giving the percentages at least basic statistical meaning. Spread out, the score sheet is stark:
- Novara (team's in-house system): composite 0.8154; DRC pass 97.5%, LVS 100%, post-sim 75%, acceptance 72.5%, area efficiency 52.2%. It was iterated repeatedly during benchmark construction—effectively a "native resident" of this question bank.
- GPT-6 Astra (OpenAI Responses API, with thinking summaries): 0.7483—the most impressive general agent. DRC 99.2%, LVS 98.3%, post-sim 82.5%, acceptance 70.8%, area efficiency 59.5%. The cost: on average 111.8 tool calls and 11.2M tokens per task.
- GPT-6.1 Sol (Codex-style, no thinking summaries): 0.7023, close behind.
- GLM-5.3 (OpenHands framework): 0.4779—but at a shocking cost of 34.5M tokens on average, three times GPT-6 Astra.
- Open-weight models: Kimi-K3 0.2900, DeepSeek V4 Pro 0.1504, Kimi-K2.6 0.0958—Kimi-K2.6's DRC pass rate was just 0.8%, essentially a blank submission.
- Nova-Silicon Team (2026). *AnalogLayoutWorld: An Open Benchmark for Agentic Analog Layout Design*. arXiv:2611.16991.
- Task sources: Analog Design Bench, LOADBench, team-owned designs, open-source designs (fixed SKY130 transistor-level netlists, 40 expert-reviewed tasks)
- Execution environment: Harbor + Docker; gdsfactory / KLayout (DRC/LVS) / Magic (PEX) / ngspice (post-layout simulation); 3-hour wall-clock budget
- Evaluation: four independent judges (DRC, LVS, PEX+post-sim, geometric quality ρ_t and aspect ratio q≤4) + morphological screen; engineering-quality bonuses for matching / critical-net routing / power delivery
- Scores: Novara 0.8154 / GPT-6 Astra 0.7483 (111.8 tool calls, 11.2M tokens) / GPT-6.1 Sol 0.7023 / GLM-5.3 0.4779 (34.5M tokens)
- Scale correlation: ρ_s = -0.73; ranging from GLM-5.3 Flash -0.82 through Kimi-K3 -0.44 to GPT-6 Astra -0.27
- Acknowledgments: Shanghai AI Lab and Fudan University
Reading the table vertically reveals a clear capability断层: Novara and the GPT family form the first tier above 0.7; GLM-5.3 holds 0.48 in the second tier at three times the token cost; below that is a cliff—open-weight models mostly struggle between 0.1 and 0.3. This stratification doesn't fully match rankings on general coding benchmarks, showing that analog layout is a task with its own capability dimensions—it requires not just writing code but understanding physical constraints, invoking specialized engines, interpreting specialized errors, and converging through multi-round iteration. Many models can write Python; few can work with ngspice and KLayout.
How to read this table? One axis is score, another is cost. GPT-6 Astra traded 11.2M tokens for 0.7483; GLM-5.3 traded 34.5M tokens for only 0.4779—more than a fivefold difference in score efficiency per million tokens. If layout agents are ever billed per call in production lines, this efficiency matters more to commercial viability than the score itself.
📉 A Correlation That Silences Everyone: The Scale Amplifier
The paper's heaviest finding hides in one correlation coefficient: task size and score are negatively correlated, ρ_s = -0.73. More devices, lower scores. Not surprising in itself—harder tasks, lower scores. But decomposed per model, the numbers turn dramatic: GLM-5.3 Flash's coefficient is -0.82, Kimi-K3's -0.44, GPT-6.1 Sol's only -0.35, GPT-6 Astra's just -0.27.
In plain language: when tasks grow from 20 devices to 60 devices, weak models fall off a cliff while strong models slide down a gentle slope. Examinees in the same room show completely different "gradients" as tasks get harder. This suggests a diverging capability spectrum: frontier models possess some engineering strategy that "stretches" with task scale—perhaps better task decomposition, more robust intermediate checkpoints, or more economical tool use—while lower-tier models may have merely memorized patterns on small tasks, exposed the moment tasks change. This gradient difference should alarm model developers more than any single score, because it determines how large a commission your model can take on.
Another finding concerns test-time scaling—one of the hottest topics in 2026 agent research: given more thinking time and tool-call budget, can a model convert budget into higher success? AnalogLayoutWorld offers a rare "physical-world" answer. GPT-family agents show a healthy behavioral pattern: bigger tasks mean more tool calls (correlations 0.66/0.68) and more time spent (0.63/0.67)—they proactively increase their own budget, like human engineers who know "this job needs two extra days." Open-weight models aren't unwilling to work overtime; their budgets hit the wall: GLM-5.3 Flash hit the 3-hour limit in 51 of 120 attempts; GPT-6 Astra hit it zero times. The capability gap shows not only in "can it" but in "does it realize it must try harder." Hitting the wall isn't the model's fault—it's the deployer's rule—but that rule is part of the benchmark, because real customers won't give you unlimited time.
🔬 Three Case Files: Reefs Above the Passing Line
Beyond scores, three appendix case studies say more about "the final judgment of physics" than any number. Their common protagonist: designs that passed functional acceptance but struck engineering-quality reefs. Judged only by "passed or not," they are all successes; only by decomposing the quality score do you see who is truly a good designer.
Case 1: The aspect-ratio trap (Task 9, ring VCO). All three of GPT-6 Astra's attempts passed DRC, LVS, and post-simulation—every functional judge stamped approval. But the geometric judge measured aspect ratios of 22.63, 26.31, and 26.24 against a limit of 4. All three attempts scored only 0.3 for being "too elongated." The cause lies in the details: the design needed a large resistor; the agent chose a straight (serpentine-free) resistor 203.19μm long, stretching the layout into a strip. Novara chose a folded type, aspect ratio 1.24, scoring 0.9015. Each of the agent's local choices was legal; together they violated a global geometric constraint—not a knowledge problem but an experience problem, and experience is exactly what textbooks don't contain. A graduate who has read every analog layout textbook would likely make the same mistake on a first independent design: they know all the resistor types but not "when to use which." The agent made the mistake a human apprentice would make; the problem is, nobody taught it.
Case 2: The invisible current (Task 17, a 30mA PMOS pass LDO). Both agents' layouts passed all functional tests, yet scores differ noticeably: 0.9580 vs 0.8418. The difference? Power delivery. GPT-6 Astra placed 462 via3 vias on the power path; DeepSeek V4.1 Flash placed 10. Each via3 has a 0.48mA current budget at 90°C—10 vias = 4.8mA capacity, while this branch must carry 30mA, needing at least 63 vias. What does a 10-via layout mean? Under long-term full load, this power path will heat, age, and possibly burn out—functional testing at room-temperature simulation detects nothing, but the laws of physics remember precisely. Power sub-scores: 0.748 vs 0.199. This is the sharpest contrast in the whole study: the difference between a design that "works" and one that "survives ten years" hides inside vias nobody sees. Human veterans always count these vias when reviewing layouts—not because textbooks say so, but because after burning a few boards, the number gets carved into your bones.
Case 3 (appendix): Right and wrong about area (OTA design). For one differential pair, two agents achieved effective device areas of 910.01 and 912.04 μm²—a 0.22% difference, near-perfect matching. But including surrounding routing and guard rings, occupied areas were 809.75 and 3575.16 μm²—more than a fourfold difference. Area scores 0.2000 vs 0.0108—area alone explains 99.4% of the total-score gap (0.9617 vs 0.7712). The brutal aesthetics of layout design on full display: your prided matching precision may be worthless next to area efficiency, and area waste is often not a technical error but an aesthetic failure to "bring yourself to cut."
🔁 Iteration Is Real Progress—Unevenly Distributed
One moderately optimistic finding: verification feedback does drive agent improvement. 90.3% of DRC first-passes and 66.8% of LVS first-passes occurred after the first verification call—agents generally "draw → verify → read errors → revise," and the loop works. That matches intuition: analog layout has always been an iterative convergence process; human designers work the same way—the difference is that humans understand every term in the error reports, know how to fix "minimum-area rule violations" and what "latch-up risk" means, while for agents facing hundreds of lines of engine output, whether they can localize the problem and generalize depends entirely on the model's depth.
But don't get too optimistic. Iteration works; not everyone can use it. Those who improve through verification feedback are mainly the head-tier agents with budget and the ability to read engine errors—Kimi-K2.6's 0.8% DRC pass rate shows that for many models, even the "draw one version" step is insurmountable, let alone iteration. The verification loop is an amplifier: the strong use it to roll a snowball; the weak lack even starting capital. This sharpens an old question: where does training data for physical-world agents come from? The lesson from CADET echoes here—analog layout may similarly need an "inductively constructed" process-data factory, letting agents master the "draw-verify-revise" loop on synthetic small tasks before entering the real exam hall.
🧭 Sober Reflection: The Court's Own Limits
Reading the table across all models reveals a fact easily obscured by headlines: the 21.3% acceptance rate is actually above 70% for head-tier models and near zero for the tail. The dire overall pass rate is dragged down by the distribution's right tail—frontier models have already crossed the threshold; the question becomes how far from "passing" to "excellence." This matters for industrial deployment: if your scenario permits top models with ample budget, analog layout agents are no longer science fiction; if you're constrained to open-weight models and limited compute, the road is long. A benchmark's averages deceive; quantiles are honest.
The authors' honesty also deserves note. The court has three known boundaries. First, task and process coverage: 40 tasks, a single process (SKY130)—generalization to advanced nodes (FinFET, GAA) and larger designs is entirely unknown; SKY130 is a mature planar process with relatively lax rules. Second, verification limits: DRC/LVS/PEX/simulation cover "legal, consistent, meets spec," but long-term reliability (NBTI, HCI aging) and robustness across PVT corners and Monte Carlo variation are not yet judged. Third, evolvability of quality assessment: the weights and thresholds of engineering-quality scoring (matching, routing, power) are expert-set and carry subjective elements—a different expert panel might slightly reshuffle rankings.
There's also a more interesting meta-question: Novara, the benchmark builder's own system, took first place—both a "native resident's" home advantage and a hidden risk of benchmark contamination—its strategies may have been tuned to these 40 tasks' distribution. The paper doesn't dodge this (reporting Novara's scores separately), but readers should keep their own scale: the real touchstone is the next challenger trained from scratch on this course.
🗺️ Closing the Trilogy
Now the trilogy's arc is complete.
Part one, the ward: AI facing viruses—an opponent it cannot converse with. It can only observe, learn, predict, then accept the verdict of hospitalization counts days later. The predictive world is probabilistic: even the best models miss 20% of the peak, and public-health wisdom converts uncertainty into contingency plans. Part two, the workshop: AI operating hands-on, every wire a promise, the DRC checker a stern foreman, errors immediately visible, punished, reworked. Part three, the final court: AI submits, graded by the laws of physics themselves, with errors silently hidden in via counts and aspect ratios until they surface years later as failures.
The ward taught it humility (short history, lean on priors), the workshop taught it discipline (data can be designed; judges must be respected), the court taught it awe (between a 91.1% submission rate and a 21.3% pass rate lies the entire world of engineering). Three exam halls, three unforgiving ways, one common theme: AI capability evaluation is migrating from "how many questions answered correctly" to "how many works accepted by the physical world." Once this migration happens, there is no going back.
Together, the three papers express the plainest and hardest sentence on the "AI doing science" narrative: AI that walks out of the exam hall must ultimately enter a world with no question-setter. And the measure of its maturity is no longer how many questions it got right, but how many of its works the physical world has accepted.