Three Fresh Agent Runs Beat One 100M-Token Session by 264 Elo

TL;DR

When Agents Slow Down is a new preprint from UC Berkeley, the University of Washington, Princeton and Bespoke Labs that measures what extra tokens actually buy a coding agent. The authors ran Claude Code, Codex, Gemini CLI and Kimi Code for up to 100M tokens per task on four open-ended benchmarks, then rated each agent's best solution so far on an Elo scale. At first, agents turn tokens into quality faster than blind best-of-N sampling. In the pooled results, though, every system's marginal gain eventually drops below that baseline. The strongest human contestants on AtCoder Heuristic Contest problems do the opposite and speed up over days. The practical payoff: on one task, three parallel sessions beat a single 100M-token session by 264 Elo and beat ten short sessions by 355.

What they measured

Standard agent benchmarks such as SWE-bench return a pass or fail verdict at the end and throw the trajectory away. That tells you nothing about how quality grows as tokens burn. So the authors used open-ended tasks where every intermediate submission gets a score from a deterministic judge, with no LLM grader and no human ratings.

There are fourteen problems from four benchmarks. FrontierCS contributes algorithmic research problems such as job-shop scheduling and two-dimensional knapsack. ALE-Bench contributes three rehosted AtCoder Heuristic Contests. MLS-Bench adds a clustering-algorithm task and a neural architecture search task. FlashInfer-Bench scores five GPU kernels, including paged MLA attention, MoE and RMSNorm, by their speedup over a reference implementation.

Four agent systems were tested: Kimi Code with Kimi K2.7, Codex with GPT-5.5, Claude Code with Opus 4.8, and Gemini CLI with Gemini 3.5 Flash. Each agent got five independent sessions per task, and each session was pushed to a 100M-token target. If an agent quit early, a watchdog resumed the same session. Token counts include cached input, since a long session spends many of its tokens re-reading its own history.

Raw scores can't be compared across tasks, so the paper uses ratings instead. The authors point to 26-circle packing: moving the score from 1 to 2 is trivial, while AlphaEvolve's move from 2.634 to 2.635 set a record. Elo only cares about which submission is better, so units and difficulty curves drop out.

The yardstick: 400 Elo per decade

The paper's first theorem gives the curve a baseline. If every attempt is an independent draw, the best of n attempts sits exactly 400 times log10(n/m) Elo above the best of m. Ten times the tokens buys 400 Elo, whatever the task. That matches the older finding from Large Language Monkeys that repeated sampling improves roughly log-linearly with compute.

The number is less arbitrary than it looks. Under the Bradley-Terry model, a 400-point gap means 10-to-1 odds. The best of ten independent attempts beats a single fresh attempt exactly 10 times out of 11. Think of hiring ten interns who never speak to each other and keeping the best homework. Any agent that remembers, plans and reads feedback should beat that.

That gives a clean test. If an agent's Elo slope is above 400 per decade, it is learning from its own history. If it is below, a fresh start would have done more with the same tokens.

Agents beat the interns, then stop

Early on, they win. Several systems exceed the 400-per-decade slope at smaller budgets, so adaptive work inside a session really does beat blind resampling for a while. Then the slopes peak and fall. In the pooled all-domain fit, every one of the four systems is below 400 by the largest budgets, and most per-benchmark curves show the same late decline. The authors say agents like Claude Code and Codex scale no faster than the sampling line "after the first few context-window compactions," and conclude that context management is still a significant problem.

More tokens still help: every system's rating keeps rising all the way to 100M. They just help less than starting over would. A sweep across four Qwen3.5 model sizes, one 27B dense model and three MoE variants, showed the same shape, so a bigger model doesn't obviously fix it.

Fancier strategies didn't escape it either. On FrontierCS Polyomino Packing with Kimi K2.7, the evolutionary methods AdaEvolve and GEPA both started strong, with AdaEvolve more than 300 Elo ahead of Kimi Code near 100K uncached tokens. By Kimi Code's last checkpoint, all three were within about 20 Elo of one another. Test-time training, following the TTT-Discover recipe on gpt-oss-20b over 6.6M rollout tokens, showed a brief stretch faster than sampling and then slid back toward the reference. Even updating the weights during the task didn't change the curve.

The humans speed up

AtCoder logs the timestamp and score of every submission, so the authors rebuilt the full histories of each contest's top 50 finishers. Pooled over seven contests that each ran at least 240 hours, both the top-10 and top-50 groups are convex in log time: later doublings of time bought bigger gains, not smaller ones. The paper reads this as evidence of continual learning within a single problem, the thing current agents don't do.

The direct comparison uses AHC014, rejudged under one local evaluator. GPT-5.6 Sol running in Codex and Opus 4.8 were rated at checkpoints up to 72 hours. Both agents led the human groups early on. Then the agents flattened, while the top-10 humans kept climbing and passed both.

The humans had eleven days, a leaderboard and, presumably, sleep. The agents had 72 hours and a context window that kept summarizing itself. The same pattern held on AHC038, AHC040 and AHC031 against all four agent systems: agents made their biggest gains early, and humans kept improving at later checkpoints.

Why long sessions get stuck

The authors offer a "sticky basin" explanation. Early in a run, the agent commits to an approach: an algorithmic idea, a code structure, a high-level strategy. Extra compute keeps polishing that approach but rarely makes the agent reconsider it, and the approach gets harder to drop as context piles up.

They back it with a toy theorem. If each session is locked into a randomly drawn basin and searches inside it, a session with a fixed multiple more budget beats a shorter one only half the time as budgets grow. The difference between basins swamps the extra depth. Picture a hiker who picks a hill in the first hour and spends three days finding its exact summit while a taller hill stands next door. Three hikers sent to three random hills beat one hiker with more days.

The authors' design suggestion follows from that: harnesses need to escape or reconsider basins, not just stretch context windows further. Problems with many plausible solution families may be especially hard to scale with one long trajectory.

Split the budget at the inflection point

The useful part is an allocation rule. Define the scaling inflection point as the last token budget where an agent's Elo slope still beats 400 per decade. Past it, if the slope stays below the line, continuing the session eventually loses to sampling fresh continuations from where it stands. So run K independent sessions, each only up to that point, and keep the best result. Another theorem says this gains 400 times log10(K) Elo over one session while keeping whatever the harness gained before the inflection. For a total budget B, K is B divided by the inflection budget, rounded.

They tested it on Polyomino Packing with Kimi Code and Kimi K2.7. The single-session inflection point landed at 38M tokens, so the rule predicted three sessions for a 100M budget. The authors then compared 1, 2, 3, 4, 5 and 10 sessions, each spending exactly 100M tokens drawn from the same pool of runs, with ratings averaged over 20,000 random session-to-group assignments.

In Bradley-Terry terms, a 264-point gap means the three-session split wins about 82% of head-to-heads against the single marathon, and a 355-point gap means about 89% against ten short runs. One session does lead at small budgets. The best session count only rises after the inflection point. The rule also beat both extremes in a replication on MLS-Bench.

What to do with this

• Score it or skip it. This only works where you can grade intermediate output automatically: optimization, kernels, performance tuning, heuristic search. Best-of-K needs a judge to pick the winner.
• Profile per task. The FrontierCS domain-level crossover did not line up with Polyomino Packing's 38M-token inflection, and MLS-Bench showed the same mismatch. Predicting the point from early-session statistics is still open.
• Treat compaction as a warning light. The flattening shows up after a few compactions, so a session that has been summarized several times is a candidate to fork, not extend.
• Build for escape. If you ship a harness, forcing a fresh look at the approach may matter more than a longer context window.
• Rerun it yourself. The code release includes a patched Harbor v0.7.1 with the token-target watchdog. The raw trial data is promised separately.

Caveats

This is a preprint and has not been peer reviewed. The headline matrix covers fourteen tasks with five sessions per agent-task pair, and the allocation win comes from one agent on one task plus one replication. The models are a generation behind the frontier: Opus 4.8, GPT-5.5, Kimi K2.7 and Gemini 3.5 Flash, with GPT-5.6 Sol appearing only in the AHC014 comparison.

The human comparison puts agent tokens and human contest hours on a wall-clock axis. Agent runs were rated only up to 72 hours, and the flat agent tails in the paper's figure are extrapolations. Bespoke Labs donated compute, and two of the authors are affiliated with it. None of that undercuts the math behind the baseline, but it limits how far the specific numbers travel.

Key Takeaways

• A real baseline. Independent best-of-N sampling gains exactly 400 Elo per tenfold compute, which finally gives agent scaling curves something to be measured against.
• Agents fade. Claude Code, Codex, Gemini CLI and Kimi Code beat that line early, but in pooled results all four fall below it by the largest budgets, after a few compactions.
• Humans accelerate. Top AtCoder contestants improve superlinearly over days; on AHC014 the top 10 reached 1,991 Elo versus 1,582 for Opus 4.8 and 1,422 for GPT-5.6 Sol.
• Split, don't stretch. Running three 33M-token sessions instead of one 100M-token session added 264 Elo on Polyomino Packing, and ten 10M sessions did worse than both.
• Harness design target. The sticky-basin result suggests the next gains come from reconsidering early commitments, not from longer contexts or evolutionary wrappers alone.

Sources: arXiv: When Agents Slow Down (2609.15309), Hugging Face paper page, Project page, Agent-TTS-Code on GitHub, FrontierCS, ALE-Bench, MLS-Bench, FlashInfer-Bench, AdaEvolve, GEPA, TTT-Discover, Large Language Monkeys, AtCoder AHC014