Meta Rebuilt 11,260 Real Coding Sessions Into a Benchmark. The Score That Matters Is How Often You Correct the Agent.

TL;DR

SWE-Together, a new benchmark from a team at Meta, throws out the one thing every coding-agent leaderboard has in common: handing the agent a fully written ticket up front and grading only the final diff. Real coding is a conversation, so SWE-Together reconstructed 109 verifiable tasks from 11,260 recorded user-agent sessions and grades agents on two axes at once. Did the repository end up correct, and how many times did you have to step in and correct the agent to get there. On the interactive tasks, Claude Opus 4.8 leads at 63% pass@1 while needing the fewest corrections, about 1.38 per task. The weakest model in the lineup lands at 40% and needs more than two. The blunt finding: the gap between coding agents is mostly a gap in how much babysitting they demand.

Static benchmarks grade a monologue. Real coding is a dialogue.

Almost every coding benchmark you have seen, SWE-bench included, is a monologue. The agent receives a fully specified ticket, works alone, and is scored only on whether the final diff passes hidden tests. That is not how anyone actually uses one of these things. You start vague, it guesses, you clarify, it drifts toward the wrong file, you pull it back, and somewhere in that back-and-forth the feature gets built. A benchmark that skips the dialogue is measuring a skill nobody exercises in isolation.

SWE-Together's bet is that the interaction is the task. It takes real sessions where a human and an agent went back and forth, freezes them into reproducible tasks, and then replays each one against a fresh agent under a simulated version of that same user. The agent does not get the tidy final spec. It gets the messy opening ask and has to earn the rest.

How you turn 11,260 messy chats into 109 gradeable tasks

The pipeline is three stages, and it is brutal. First a deterministic filter drops any session without multiple user messages, real code edits, and repository context. Then an LLM viability screen asks a harder question: can this work be rebuilt as something reproducible and verifiable, and it rejects anything that leans on external state like a live pull request, a deployment, or credentials. Finally the survivors are reconstructed in an isolated sandbox, cloning the repo at a pinned commit and writing out the task. Only 109 made it through, under 1% of the raw pile. Verifiable, interactive coding tasks are apparently rare enough to be worth mining thousands of chats for a handful.

The clever part is the user. SWE-Together builds a reactive LLM simulator that stands in for the original human, and it is trajectory-conditioned rather than scripted. It watches what the agent just did and only then decides whether to speak. Think of it as a customer reading over your shoulder who stays quiet while you are on track and only clears their throat when you start walking toward the wrong file. It has five moves: stay silent, ask a question, redirect, add a new requirement, or tell the agent to go check something. Because it reacts to the live trajectory instead of firing on a schedule, two different agents on the same task get feedback at different moments, exactly like two real pairing sessions would diverge.

The scoreboard, and the number that actually matters

Seven frontier agents ran the gauntlet, each twice. On final correctness (pass@1, meaning the repo satisfies the task's weighted rubric), Opus 4.8 tops the board, GPT-5.5 and the older Opus 4.6 tie just behind, and the open-weight GLM-5.2 is a hair back from them. DeepSeek-V4-Pro and MiniMax-2.7 bring up the rear.

But pass@1 is only half the report card, and it is the boring half. The other half is User Correction: how many times the simulated user had to intervene to keep the agent on course. And it tracks capability almost perfectly. The Pearson correlation between correction count and pass@1 is negative 0.92, close to lockstep. The better an agent scores, the less you have to touch it. That is the paper's real contribution: it turns "it just gets it" from a vibe into a measurable axis.

One number to sit with before you crown a winner. GPT-5.5 nearly matches Opus 4.8's score using less than half the tokens (about 29.9k versus 74k per task) and a third of the wall-clock time. So "best" depends on whether you are paying for the answer or paying for the meter.

The answer key fails its own exam

Here is the detail that reframes the whole scoreboard. The human reference patch, the actual solution a real developer shipped, scores only about 78%, not 100%. The reason is honest and a little funny: roughly 35% of a task's graded goals are process requirements, things like running the tests or following a repo convention, that a final diff simply cannot express. So they dock every submission equally, the human answer key included. The answer key fails its own exam by about 22 points.

The practical read is to stop measuring against a perfect 100. Treat 78% as the real ceiling on this benchmark. That puts Opus 4.8's 63% about 15 points off what a human actually achieved, not off some unreachable ideal. The room to improve is real, but it is smaller than the raw number suggests.

Did you notice the user was a bot?

The entire benchmark rests on the simulated user being believable. If a human grader could smell the bot, the scores would be theater. So the authors ran a Turing test: shown a mix of real and simulated user turns, people identified the simulator correctly only 46% of the time, which is below a coin flip. The fake user is statistically indistinguishable from a real one. Whether that is a triumph of simulation or a quiet comment on how much real code review already reads like being nagged by a script, the paper does not say.

Just as important, the simulator's fidelity (its Intent Coverage) held steady between 0.68 and 0.72 across every agent. In plain terms, the stand-in user was not secretly easier on the strong models than the weak ones, so the head-to-head numbers are actually comparable.

What this means for you

• If you evaluate agents. Stop reading final pass rate in isolation. Two agents can land the same diff while one needed you three times and the other needed you once, and only the second one is worth putting in front of your team. Correction burden is the axis that predicts whether people will keep using the thing.
• If you buy or route models. The token and time gap is enormous. GPT-5.5 delivering near-top quality at a third of the cost per task is the kind of number that decides an inference bill, not a leaderboard tie.
• If you build harnesses. This benchmark rewards agents that recover gracefully from a mid-course correction, not just ones that nail the first shot. Design your loop so a redirect is cheap and the agent does not throw away its progress every time the user changes their mind.
• If you have your own session logs. The recipe (mine real sessions, screen for what can be rebuilt, replay under a reactive simulator) is a template you can steal to grade agents on your product's traffic instead of someone else's tickets.

The honest caveats

Keep the enthusiasm proportional. 109 tasks is a small set drawn from four upstream session sources, so a few skewed repos could move the numbers. The simulator is an LLM judging and prompting other LLMs, which carries the usual circularity risk even with a 46% Turing pass rate. The process-requirement penalty that caps the reference patch at 78% is a modeling choice, not a law of nature, and reasonable people could weight those goals differently. And with two runs per task, these are directional signals, not settled rankings. Useful, not gospel.

Key Takeaways

• SWE-Together is the first widely noted coding-agent benchmark that grades the interaction, not just the final diff, rebuilding 109 verifiable tasks from 11,260 real user-agent sessions.
• Claude Opus 4.8 leads at 63% pass@1 and needs the fewest corrections (about 1.38 per task); the field trails down to 40%.
• The headline result: correction count and final score move together at negative 0.92 correlation, so a coding agent's real quality is mostly how little you have to steer it.
• The human reference patch only scores about 78% because a third of the graded goals are process requirements no final diff can express, so treat 78%, not 100%, as the ceiling.
• Cost matters: GPT-5.5 nearly matches the leader at roughly a third of the tokens and time, so the cheapest good-enough agent may beat the highest scorer in practice.
• The methodology (real sessions plus a reactive user simulator that passed a Turing test) is a reusable blueprint for evaluating agents on your own traffic.

Sources: SWE-Together (arXiv:2606.29957), SWE-Together full paper, AI News Today, July 6 2026