A New Benchmark Grades Coding Agents Like Senior Engineers. The Best One Passes 24% of the Time.

TL;DR

Senior SWE-Bench, a new evaluation from Snorkel AI built with researchers from Princeton and UW-Madison, asks a rude question: what happens if you grade a coding agent the way you would grade a senior engineer? Vague ticket, multi-service feature, and a reviewer who cares whether the patch fits the codebase, not just whether tests go green. The answer: Anthropic's Claude Opus 4.8 leads at 24.0% pass@1, GPT-5.5 manages 16.0%, and Claude Sonnet 5, which shipped on June 30, is already on the leaderboard at 19.4%. The benchmark went up in mid-June and hit the Hacker News front page this week with those fresh scores. If you have been wondering why 77% on SWE-bench Verified does not feel like 77% in your repo, this is the cleanest data yet on the gap.

The Old Bar Was a Parking Lot. This Is Rush Hour.

The original SWE-bench and its Verified subset did their job: they proved agents can resolve real GitHub issues. But they are close to saturated. Even Qwen3.6-27B, an open-weight 27B dense model you can run on a single GPU, reports 77.2 on SWE-bench Verified. When a model that fits in 17GB of VRAM posts a number like that, the benchmark has stopped separating the field.

Senior SWE-Bench, whose site credits Karthik Narasimhan (a co-author of the original SWE-bench) and UW-Madison's Fred Sala, changes what is being measured, not just how hard the tasks are. SWE-bench Verified is a driving test in an empty parking lot: clearly marked cones, one maneuver, pass or fail. This benchmark merges you onto the highway at rush hour and grades your lane discipline the whole way.

Tickets Written Like Tickets, Not Like Exams

The tasks come from real pull requests merged after February 2026 in open-source repositories, mostly authored by engineers with 100+ commits in that repo, and half the repos are less than five years old. There are 100 tasks in two flavors: Design-and-Build (features and migrations) and Investigate-and-Fix (bugs and performance issues). Fifty tasks are public, fifty are private to resist contamination, and the post-February-2026 cutoff means the reference solutions postdate most training data anyway.

The instructions are the interesting part. Instead of a lovingly over-specified prompt, feature tasks read like what a product manager would actually hand you: an overall request plus behavioral requirements. Bug tasks read like incident reports ("it looks like the PG replication slot lag is growing with no bound on prod stacks"). The methodology post shows an equivalent task specified in 6,008 characters and roughly 39 code symbols on SWE-bench Pro, versus 639 characters and zero code symbols here; the median instruction is about a third the length of SWE-bench Pro's. Feature tasks touch an average of 11 files and take even the strongest agents hundreds of steps.

What Counts as a "Tasteful Solve"

Passing tests is necessary but not sufficient. A submission only counts as a tasteful solve if it survives the full gauntlet: all pre-written verifiers and adaptive validation tests pass, a quality rubric scores above 0.5, patch bloat stays under 2x the reference implementation's lines of code, and a codebase-practice score clears 2 out of 5. A separate taste judge grades minimality, approach quality, hygiene, pattern adherence, and whether the code fits the surrounding abstractions. Agents love to ship four times the code the job needed; the bloat gate caps that enthusiasm at two.

The adaptive validation is the clever mechanism, because under-specified tasks mean the agent's interface can legitimately differ from the reference PR. A validation agent (built on Mini-SWE-Agent with Claude Sonnet 4.6) writes behavioral tests from expert-designed recipes, adapted to whatever interface the submission actually built, and an LLM judge reviews the result. Each validation recipe has to pass three times on the oracle patch and consistently fail on no-op patches before it is trusted. All of that machinery adds a median of 11% wall-clock time and 6% token cost per run, which is cheap for a grader that cannot be gamed by hardcoding the expected function name.

The Capability Is There. The Judgment Is Not.

The most useful finding is buried in the methodology: when the team ran guided variants of the tasks, with diagnostic hints included, Claude Opus 4.8 and GPT-5.5 both solved 85-90%, and all but 8% of tasks were solved by at least one agent. So the raw coding ability to complete this work exists today. What collapses the score from roughly 90 to 24 is everything senior engineers are actually paid for: deciding what the vague ticket means, finding the right place in an 11-file blast radius, and not bolting a fourth abstraction layer onto a codebase that has three.

That gap is the practical takeaway for anyone running agents in production. The model is not the bottleneck; the specification and the review gate are. Which also explains the leaderboard order: Sonnet 5, two days old, already outscores GPT-5.5 here, and open-weight GLM-5.2 lands at 12.5%, about half of Opus 4.8, with Kimi K2.6 at 8.2% and Gemini 3.5 Flash at 3.0%. If your local model writes working-but-ugly patches, this benchmark finally has a number for the ugly.

Run It Yourself, and the Caveats

The 50 public tasks are on GitHub as a dataset for the Harbor eval framework, which can read the repo directly: point harbor run at snorkel-ai/senior-swe-bench-v2026.06 with your agent and model of choice. That makes it one of the few senior-tier evals you can run against a self-hosted model tonight.

Now the straight-faced part. This is 100 tasks, so single-digit gaps between neighboring models are noise. The taste judge is itself an LLM with deliberately conservative thresholds, and "quality" judged by a model deserves the same skepticism as code reviewed by one. Half the tasks are private, which fights contamination but means you are trusting the maintainer's grading. And Snorkel is a company that sells evaluation and expert-data services; a benchmark showing that everyone needs better evaluation is not exactly off-message for them. None of that invalidates the core result, which matches what every team running coding agents already feels: the distance between "resolves the issue" and "I would merge this" is where the next two years of agent progress live.

Key Takeaways

• Senior SWE-Bench grades agents on under-specified, multi-file tasks with code-quality gates; Claude Opus 4.8 leads at 24.0% pass@1, with Claude Sonnet 5 at 19.4% and GPT-5.5 at 16.0%.
• A "tasteful solve" must pass behavioral tests, a quality rubric above 0.5, patch bloat under 2x the reference, and codebase-practice alignment above 2/5.
• Instructions are realistic: a median around a third the length of SWE-bench Pro's, roughly zero code symbols, and feature tasks averaging 11 files touched.
• With diagnostic hints, the same agents solve 85-90% of tasks, so the failure is judgment under ambiguity, not raw coding ability.
• Tasks come from post-February-2026 PRs and half the set is private, making contamination much harder than on classic SWE-bench.
• The public half runs via the Harbor framework against any model, including self-hosted open weights like GLM-5.2 (12.5%) and Kimi K2.6 (8.2%).

Sources: Senior SWE-Bench leaderboard, Senior SWE-Bench methodology post, senior-swe-bench-v2026.06 on GitHub, Anthropic: Claude Sonnet 5, Qwen3.6-27B model card