GPT-5.2 Went 21.8% to 41.2% Without Touching the Model

TL;DR

SWE-Bench ProMax, posted August 10 and accepted at COLM 2026, is a 170-instance refactoring benchmark built from real commits in seven languages. The best configuration tested, GPT-5.2 under OpenHands, resolves 41.2%. The same model under mini-SWE-agent resolves 21.8%. Nothing about the weights changed. If you build with agents, that spread is the actual headline, and open-weight contenders reach 36.5% at a fifteenth of the cost.

Why anyone needed another SWE benchmark

SWE-bench Verified was the number everyone quoted, right up until it stopped meaning anything. OpenAI stopped reporting against it on February 23, 2026, and published why: it audited the 138 instances its o3 model could not solve consistently across 64 independent runs, put at least six experienced engineers on each case, and found 59.4% of them materially flawed.

The failures were not close calls. 35.5% of the audited tasks carried tests that enforced a specific implementation, so a functionally correct patch that chose a different structure was scored wrong. Others tested requirements the issue never stated. OpenAI also reported that frontier models could reproduce gold patches verbatim, which is the polite way of saying the answer key was in the training set.

The ProMax authors take that audit as their starting premise and go somewhere else entirely. Instead of patching Verified, they changed the task.

Refactoring is a harder ask than bug-fixing

A bug fix is local. A refactor is not: it demands coordinated, behavior-preserving edits spread across a codebase, where the tests that pass before must pass identically after. Fixing a bug is replacing a broken light switch. Refactoring is rewiring the house so every switch still turns on exactly the light it did before, with the inspector watching.

That difference shows up in the numbers. The paper reports that 86% of SWE-bench Verified instances modify only a single file. ProMax instances average 11.4 modified files and 261.6 lines of code, with 30% touching more than ten files and 32% requiring over 200 lines.

The curation is the expensive part. Issue descriptions were rewritten from scratch to be unambiguous specifications, test suites were manually reviewed to strip both the overly narrow and the overly broad, and anything without real cross-file scope was filtered out. The 170 survivors span 70 repositories: 29 Python, 28 TypeScript, 26 Java, 23 Go, 22 C++, 22 Rust, 20 C.

The scores, and the thing hiding inside them

Six models were run under two scaffolds. Three proprietary (Gemini-3-Pro, Claude Sonnet 4.6, GPT-5.2) and three open-weight (GLM-5, Kimi-K2.5, Qwen3.5). Read the pairs, not the leaderboard.

GPT-5.2 nearly doubles, from 21.8% to 41.2%. Qwen3.5 jumps 16 points. GLM-5 gains almost 14. And Gemini-3-Pro goes down, 26.5% to 19.4%, which is the most relatable finding in the paper: give something more tools and more autonomy and it can absolutely make things worse.

This is the practical takeaway for anyone shipping an agent. The scaffold is not neutral plumbing. It decides how many files the model sees at once, how it edits, when it runs tests, and how long it is allowed to keep going. A model that thrives in a rich loop can drown in a minimal one, and the reverse holds too. Benchmarking a model without naming the scaffold is like timing a driver without mentioning the car.

The open-weight models are not the story you expect

The gap at the top is 4.7 points between GPT-5.2 at 41.2% and GLM-5 at 36.5%. The gap in the bill is not 4.7 of anything.

GLM-5 costs $0.24 per instance against GPT-5.2's $3.60 and Sonnet 4.6's $4.77. The paper's own summary is blunt about it: higher cost does not translate to proportionally higher resolve rate. Gemini-3-Pro paid $1.49 an instance to finish last.

If you are running a refactoring agent over a large repo at volume, that ratio is the whole economic argument. Roughly 89% of the top score for about 7% of the spend is a trade a lot of pipelines should be taking, especially where a failed attempt is cheap to discard and retry.

Caveats worth holding onto

• 170 instances is small. A single resolved task moves the score about 0.6 points, so treat 36.5% and 38.8% as roughly the same tier rather than a ranking.
• Only two scaffolds were tested. The scaffold effect is real and large, but mini-SWE-agent and OpenHands do not span the space. Your harness may land anywhere.
• Contamination is deferred, not solved. ProMax draws from real commits, and real commits eventually become training data. Fresh curation buys time, not immunity.
• Model versions are a snapshot. These are the checkpoints available when the runs happened, and every one of them will be superseded.

Also, yes, we have now reached SWE-bench ProMax, which means benchmarks are being named on the same schedule as phones. Somewhere a Pro Ultra Max is already in curation.

Key Takeaways

• SWE-Bench ProMax puts agents on 170 curated multi-file refactors across seven languages, drawn from 29,782 candidate commits in 70 repos.
• The best result is 41.2% (GPT-5.2 under OpenHands), so the benchmark is nowhere near saturated.
• Scaffold choice swung GPT-5.2 from 21.8% to 41.2% and dropped Gemini-3-Pro from 26.5% to 19.4%. Never quote a model score without naming the harness.
• GLM-5 hit 36.5% at $0.24 per instance versus $3.60 for the leader, so cost and capability have decoupled hard at the top of the open-weight tier.
• The benchmark exists because OpenAI's own audit found 59.4% of the SWE-bench Verified instances it examined were materially flawed, and it stopped reporting the metric on February 23, 2026.
• Tasks average 11.4 files and 261.6 lines changed, against 86% single-file instances in Verified. Multi-file coordination is where current agents actually break.

Sources: arXiv 2608.09802 (SWE-Bench ProMax), full paper and results tables, Hugging Face paper page, SWE-Bench ProMax dataset, OpenAI: Why we no longer evaluate SWE-bench Verified, OpenHands, mini-SWE-agent, SWE-bench