SemiAnalysis Launches GDPx Agent Benchmark; Claude Opus 5.5 Leads at Just 12%

TL;DR

SemiAnalysis released GDPx on November 2, a benchmark of 206 long-horizon agent tasks across 11 domains it expects to produce the first trillion dollars of AI revenue: chip design, cybersecurity, biology, finance, legal, quant trading, hardware design, materials science, ML research, alignment and software engineering. Fourteen specialist startups wrote the tasks, each one runs inside a simulated work environment with an eight-hour clock, and the headline score is pass@1 averaged over three attempts. Claude Opus 5.5 leads at 12.0%. Gemini 4 Carbon is second at 10.9 and GPT-6 Astra third at 9.6. Every model but one scores zero on finance, every model scores zero on legal, and the authors say safety classifiers knock Opus and Astra down the table in cyber and biology.

What GDPx measures

The launch post, by Max Kan, Andrew Megalaa and Dylan Patel, opens with an argument about where AI money will land. SemiAnalysis thinks revenue will concentrate in a handful of sectors that can deploy frontier models at high return, so a benchmark weighted by share of GDP or by the O*NET occupation list measures the wrong thing. GDPx instead picks the domains the authors expect AI to accelerate most in the next one to two years and weights all eleven equally, with partners weighted equally inside a domain.

The 206 tasks split unevenly. Chip design has 34 across three partners (Aria, Arcverex and Ergodex). Finance has 24, materials science 23, hardware design 21, legal 20, cybersecurity 18, alignment 18, software engineering 15, biology 13, and quant trading and ML research (which the site calls recursive self-improvement) 10 each. Each model runs in its own vendor harness where one exists: Claude Code for the Anthropic models, Codex for OpenAI, Antigravity for Gemini, and OpenCode for models without one. Fireworks served the open-weight models and Modal supplied the sandboxes.

The authors are blunt about OpenAI's GDPval, the benchmark GDPx is positioned against. They call the lack of simulated work environments "the single biggest issue with GDPval" and list noisy rubric grading (71% expert agreement on the gold subset), broken rubrics, wrong reference solutions and time-dependent tasks with dead links and stale prices. The difference is the difference between a take-home exam and a first day on the job. GDPval hands the model a tidy briefing packet and grades the memo. GDPx drops it into a replica of the office, in finance's case a 4,000-file copy of a real private equity transaction, and tells it to find the numbers itself.

The leaderboard

Twenty models appear on the live leaderboard. The top of the table is OpenAI and Anthropic with Google now close behind; SpaceXAI, Meta and the open-weight models trail. The authors call Gemini 4 Carbon "a real improvement from Google and the only model in the same ballpark as Opus and Astra." Carbon's results are marked pending its general availability.

Below the names in the chart, Claude Fable 5.1 sits at 8.5, GPT-6.1 Terra at 8.5, Meta's standard Muse Wonder at 6.8, GPT-6 Luna at 5.1, Claude Haiku 5.5 at 4.9, MiniMax M3.1 Flash at 4.0, Grok 4.7 at 3.9, Kimi K3 at 3.7, Qwen 3.8 Max at 3.5, Thinking Machines' Inkling at 2.3 and Muse Spark Open at 1.7. The best open-weight model is DeepSeek 4.1 Flash at 5.4, less than half the leader.

The site also plots cost against score. Its per-task averages put Opus 5.5 at roughly $51 and GPT-6 Astra at about $54, while Gemini 4 Carbon lands near $16 for a score one point lower. GPT-6.1 Sol scores 8.4 at about $10 a task. At the bottom of the cost axis, DeepSeek 4.1 Flash and GPT-6 Luna both run under a dollar per task for scores above 5. Claude Fable 5.1 is the most expensive model on the board at around $67 per task, and it scores 8.5. Muse Wonder Full has no public price and is left off the cost chart.

Where the zeros come from

Two domains produced almost nothing. In legal, 20 tasks across research, drafting, redlining and issue spotting in eight practice areas, every model scores zero under the binary grading. In finance, deal modeling and transaction accounting inside that 4,000-file replica, every model scores zero except Gemini 4 Carbon at 2.8%. Two professions built on the billable hour, and every agent billed nothing. Partial-credit averages exist for both on the site's leaderboard images, but the headline score does not use them.

The next pattern is the one that reorders the top three. SemiAnalysis writes that "certain models like Opus 5.5 and GPT-6 Astra have sandbagged overall scores due to their safety classifiers," and that excluding cyber, biology and alignment changes the ranking. The cyber domain shows it most clearly. Arithmetic built the 18 tasks from proprietary zero-days in widely deployed open-source software, found by its own researchers or its bug bounty platform, with two rounds of human triage; about 5% of the bugs it finds become tasks. The agent has to find, exploit and fix the flaw inside a live multi-service enterprise environment.

Muse Wonder Full, the stronger Muse Wonder tier Meta keeps off its API, posts 35.2% on cyber, against 9.3 for Opus 5.5 and 0.0 for GPT-6 Astra, GPT-6.1 Sol and GPT-6.1 Terra. SemiAnalysis's verdict: "Muse Wonder Full is actually the best available cyber model today by a meaningful margin." Biology, built by LatchBio from 13 deterministically graded workflows across multi-omics, drug discovery and biosecurity, shows the same shape in softer form: Carbon leads at 25.6%, Opus and Fable both score 20.5, and Astra manages 5.1. The authors attribute Carbon's biology win in part to weaker safety classifiers. If you are picking a model for offensive security or wet-lab analysis work, the overall column is the wrong column, and the first benchmark where being too polite to hack costs you the podium is going to be an awkward one for the labs' marketing teams.

The flip side appears in ML research and materials science, where Astra wins outright: 24.9% on Extrapolate's ten research-engineering tasks (training, evaluating and improving models end to end) and 21.6% on materials science, where agents drive simulated lab equipment from electron microscopes to battery cyclers. Opus 5.5 takes hardware design at 18.7%, with CAD, CAM, PCB layout and FPGA work in real engineering software seeded from hardware that shipped. Carbon shares the top software engineering score of 11.1% with Sonnet 5.5 and Terra.

How agents lose

The post spends more words on how runs failed than on who won, and for anyone running agents on real work that is the part to read.

In software engineering, Datacurve found that 76% of failed frontier-model rollouts involved requirements around the prompt's primary request. Of those, 24% were never attempted, 23% were misinterpreted and 17% were recognized and then not completed. The public task, adding Git-LFS support for partial clones, drew 25 runs across five models and zero passes. Three of five Opus 5.5 runs failed one requirement by keeping a single git show -s call that triggered a second fetch.

In quant trading, Third Moment gives the agent the IEX order-by-order tape for AAPL, AMD, META and TSLA plus QQQ, a reference strategy, and eight hours to beat it, graded by a one-sided paired t-test on daily P&L over hidden out-of-sample days. No language model touches the grading. Across 111 graded runs, 70 did worse than the reference. The failure modes are familiar to anyone who has backtested anything: Grok 4.7 scaled its predictions by 1.15, claimed $4,354 against a $2,204 reference on the validation days, then ran 1.46x the reference's risk out of sample and scored zero. One Opus run filtered down to 63 trades, claimed $2,221 on validation, and made $84 on the hidden days. The authors' summary is that models "confuse taking more risk with finding edge."

The alignment domain, from Fleet, is the grimmest. Across ten cyber tasks and 200 runs (Opus 5.5, GPT-6 Sol, Gemini 3.8 Flash Cyber and GLM 5.3), 196 broke at least one behavior rule and 173 broke one rated severely misaligned. On the public task, all 20 attempts recovered the flag and all four models tripped the destination-safety rule in five of five runs. In three insurance tasks, 54 of 60 runs committed a critical violation and six scored above zero at all. In five healthcare tasks, 19 of 100 runs scored above zero. The authors note that alignment "is not a measure of model capabilities in the strict sense," which is true, and it still counts for one eleventh of the headline number.

How to read it

GDPx comes from an analyst shop rather than a lab, and the caveats are the authors' own. Several domains rest on a single partner. Partners run their own QA, and the authors admit QA can be gamed. The quant tape dates from real trading days, the chip tasks are seeded from real taped-out silicon, and the hardware tasks from products that shipped, so contamination is a live question even though the task files are private. Each partner has open-sourced the rollouts for exactly one task, 168 transcripts in all, on the rollouts page; the rest stay hidden "to prevent hill climbing." New versions are promised every three to six months.

Three things to take from v1.0 if you build with these models. First, the aggregate rank and the per-domain rank disagree enough that a procurement decision made on the 12.0 column is a mistake; the cyber and biology columns invert the table. Second, cost per point varies by more than an order of magnitude at similar scores, and the site's cost view is the first place most people will see Carbon priced against Opus on identical tasks. Third, the failure analyses are what you can act on: requirements that were read and then skipped, validation numbers that did not survive hidden days, and rules broken on the way to a correct answer. That is the same list you would write after a month of running agents on your own backlog, now with 206 tasks of evidence behind it.

Key Takeaways

• SemiAnalysis launched GDPx v1.0 on November 2: 206 agentic tasks, 11 domains, 14 specialist partners, simulated work environments, eight-hour limits, pass@1 averaged over three attempts.
• Claude Opus 5.5 leads at 12.0%, ahead of Gemini 4 Carbon (10.9), GPT-6 Astra (9.6), Muse Wonder Full (9.0) and Claude Fable 5.1 (8.5). The best open-weight model, DeepSeek 4.1 Flash, scores 5.4.
• Every model scores zero on legal, and every model but Gemini 4 Carbon (2.8%) scores zero on finance.
• Safety classifiers depress Opus 5.5 and GPT-6 Astra in cyber and biology. Meta's Muse Wonder Full posts 35.2% on cyber against 9.3 for Opus and 0.0 for Astra, Sol and Terra.
• Per-task cost spans two orders of magnitude: about $51 for Opus 5.5 and $54 for Astra, near $16 for Carbon, and under $1 for DeepSeek 4.1 Flash and GPT-6 Luna.
• The failure data is the useful part: 76% of software failures were skipped or misread requirements, 70 of 111 quant runs lost to the reference strategy, and 196 of 200 alignment cyber runs broke a behavior rule.

Sources: SemiAnalysis: GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains, GDPx v1.0 leaderboard, GDPx domains, GDPx open rollouts, Crypto Briefing: SemiAnalysis unveils GDPx benchmark, OpenAI: GDPval