31 Models Designed Real Circuits. The Capacitors Fought Back.

TL;DR

On September 4 the team behind atopile published EEBench V1, a benchmark that hands a model 13 electrical engineering tasks, takes back a circuit written in code, and grades it by running SPICE against real manufacturer part models at worst-case tolerance corners. No human graders. No LLM judge. The top score on the board is 69.3%, from GPT-6 Astra, which landed after the launch post went up. Claude Opus 5 sits at 61.6%, Grok 4.6 at 57.1%. The ranking is not the interesting part. The interesting part is how they fail: one submission specified a 22 microfarad capacitor for a job that needed 545 microfarads of effective capacitance, and the part delivered 11.4 once there was voltage across it.

Why the circuit lives in code

OpenAI put a hardware demo on the front of the Astra launch page: the model taking a schematic and laying out a manufacturable board in KiCad, placing parts and routing copper, shown as a condensed clip of a run listed at 2 minutes 54 seconds. That demo is what prompted the EEBench post, and it also explains the design choice underneath the benchmark.

An agent driving a graphical CAD tool burns most of its context on coordinates, menus and application state. atopile puts the circuit in declarative code instead, so the agent edits components, nets and electrical constraints directly, builds, simulates, and reads what failed without leaving the project. It is MIT-licensed Python with roughly 3,700 stars, and it is the substrate the whole benchmark runs on.

This is similar to giving a coding agent a compiler and tests, except the tests are measuring voltages and component behavior.

That framing is EEBench's own, and it is the right one. Every check is deterministic: build the submission, construct the circuit graph and bill of materials, generate the SPICE decks, bind measurements to named probes, compare each one against a limit.

The capacitor that was not there

The one public sample task is a residential energy meter. When the 5 V auxiliary supply collapses, the processor needs 20 ms to commit its accumulated kWh reading, and the protected rail has to stay above a 3.0 V brownout floor for that whole window. The Schottky ORing path, the 93 ohm load and the 2 ohm recharge source impedance are fixed. The model designs only the MLCC reservoir bank.

Every model gets the concept immediately: add a capacitor. Then reality arrives. A ceramic capacitor's printed value is measured at zero volts, and it quietly loses a large fraction of that value once you put a DC bias across it. Think of a suitcase that shrinks the more you pack into it. The rating on the label was never a promise about the condition you actually use it in.

The submission EEBench published specified 22 microfarads nominal, X5R dielectric, 0805 package. At 4.7 V bias the grader measured 11.4 microfarads of effective capacitance, against a 545 microfarad requirement. The source built cleanly. The rail fell below 3 V after 0.85 ms.

Nothing in that failure is a reasoning error in the way we usually mean it. The model wrote valid code, picked a real orderable part, and produced a design that built without complaint. It just did not survive contact with a datasheet curve.

The board itself

EEBench lists 31 models across 10 providers, 13 tasks each, with most models logging 39 attempts, three per task. Anthropic models run inside Claude Code, OpenAI's newest inside Codex CLI, Grok inside Grok CLI, the long tail through OpenRouter.

The September 4 blog post says, in plain text, that the team had no GPT-6 Astra result yet and would like one. The leaderboard data file the site now serves is stamped the same day and has Astra sitting on top of it. Fast turnaround, and a reminder that a live benchmark page and a blog post about that benchmark page can disagree by a few hours.

One caveat before anyone quotes the ordering as settled. Thirteen tasks is a small corpus. EEBench publishes a standard error per model, and Astra's is 10.7 points against Opus 5's 6.0. The 7.7-point gap at the top of the table is comfortably inside that noise. What the table supports is a band, not a podium.

Two different ways to score around 60

The per-task breakdown is where the leaderboard earns its keep, because two models with similar headline numbers got there in completely different shapes.

Astra scored a clean 100% on 7 of the 13 tasks, took a zero on one, and split the remaining five. Opus 5 never scored 100% on anything and never scored zero on anything either, landing every single task somewhere between 16.3% and 89.2%.

For most benchmarks that difference is trivia. For hardware it is the whole thing. A rail that is 80% correct is a rail that browns out. If you are choosing a model to close a requirements loop unattended, the one that either finishes the spec or visibly falls over is easier to build a gate around than the one that hands you a plausible partial every time.

Effort is not skill

The cost and token columns are the most quietly damning part of the table.

Astra averaged 5,967 output tokens, 17 steps and 106 seconds per task at a listed $0.83. Opus 5 averaged 49,030 tokens, 26.5 steps and 781 seconds at $2.38. Grok 4.6 averaged 126,635 tokens, 72.8 steps and 2,126 seconds at $2.11, for a score 12 points lower. Claude Opus 4.8 Max spent 2,159 seconds and $5.73 a task to land at 51.4%. EEBench flags the Astra and Grok cost figures as estimated, so treat those two as directional.

The bottom of the table has its own lesson in value for money. Mistral Medium 3.5 scored 5.9% at $5.27 a task, which makes it the single most expensive way on the board to not design a circuit.

What the number actually measures

Worth knowing before you cite the percentage anywhere. The published methodology says the leaderboard score is 0.65 times a technical score plus 0.35 times cost efficiency, with the submitted bill of materials priced against a reference design at quantity-100 distributor pricing. Cost credit only counts once the circuit works, so you cannot game it with a cheap board that browns out.

• Held out on purpose. Tasks are original, authored in 2026, and never published, including the hidden tolerance limits. Nothing in the corpus should be in a training set.
• Simulation only. V1 covers the requirements, design and test loop. Layout, fabrication and physical bring-up are explicitly out of scope, with V2 slated to add them.
• Reasoning is per model. Reasoning-capable models run at high effort, but the exact configuration is chosen per model, which is a real comparability caveat.
• Harder than the sample. The public example is an easy-tier task. Harder ones ask the agent to synthesize a multiple-feedback low-pass filter around an op-amp, solve the resistor and capacitor ratios for the required poles, and hold gain, cutoff and Q inside limits after every part is pushed to a worst-case corner.

Who is paying for this

EEBench is built and funded by atopile, which sells the design tooling the benchmark runs on and says it also works with frontier labs on larger evaluation suites and simulation-backed RL environments. They state they pay the API spend themselves and sell tooling rather than scores. That is a clean disclosure, and it is still a benchmark run by a vendor whose product is the interface being measured.

The useful outside signal is that a lab picked it up independently. xAI included EEBench in the Grok 4.6 model card, in a section on engineering acceleration alongside 3D modeling and parametric CAD, publishing 60.0% at xhigh reasoning effort and 53.0% at high. EEBench's own harness puts Grok 4.6 at 57.1%. Two independent runs, three points apart, is about as close to reproducibility as new benchmarks usually get.

Key Takeaways

• EEBench V1 grades AI circuit designs with SPICE against real manufacturer parts at worst-case tolerance corners. No human graders, no LLM-as-judge, 13 held-out tasks, 31 models.
• GPT-6 Astra leads at 69.3%, ahead of Claude Opus 5 at 61.6% and Grok 4.6 at 57.1%. With a 10.7-point standard error on the leader, read the top of the table as a band, not a ranking.
• The characteristic failure is physical, not logical: a 22 microfarad ceramic part delivering 11.4 microfarads under DC bias against a 545 microfarad requirement, in code that compiled fine.
• Astra hit 100% on 7 of 13 tasks and zero on one. Opus 5 never hit either extreme. For hardware gating, that shape matters more than the average.
• Token spend does not buy accuracy here. The top score used 5,967 output tokens per task against 126,635 for a model 12 points behind it.
• The benchmark is funded by the vendor of the design tool it runs on, and stops at simulation. xAI's independent 60.0% on the same suite is the strongest external check available so far.

Sources: EEBench, Can AI design circuit boards yet?, EEBench V1 leaderboard, EEBench methodology, xAI Grok 4.6 model card, atopile on GitHub, OpenAI, GPT-6 Astra