Sakana's MASS Self-Improvement Loop Lifts Research Agents 1.2-1.6x, Slips on SWE-bench

TL;DR

Sakana AI and UC Berkeley published Recursive Self-Improvement through Multi-Agent Self-Supervision (MASS), with code on GitHub and a write-up that hit Hacker News this weekend. One Qwen3.6-27B model plays executor, optimizer, and judge: an inner loop evolves a multi-agent team workflow with the weights frozen, then an outer loop fine-tunes a LoRA adapter on the best team's transcripts with the workflow text deleted. After two cycles, score per output token on four open-ended research benchmarks rose 1.2-1.6x over the base model, the held-out win rate against the base model reached 69.9%, and the model's own judgments agreed with external judges 93% of the time, up from 73%. Transfer is uneven: SWE-bench Verified and Terminal-Bench 2.0 drifted down a point or two. The headline is "recursive self-improvement." The fine print is "on research-style tasks, for two cycles, with a 27B model and six H100s."

What MASS actually does

The paper's bet is that the cheapest way to make a model better is to let it redesign its own team first. A "workflow" here is a plain-text specification of an orchestrator and its subagents: each subagent gets a role, instructions, and an output contract, plus the call order between them. The workflow gets appended to the task prompt and executed with qwen-code 0.20.0 as the coding-agent runtime.

Two loops alternate:

• Inner loop (weights fixed): the current model proposes an edit to the workflow, runs it on the task, and compares the output against the best-so-far output, acting as its own judge. The search keeps one elite workflow at a time. The optimizer is told to prioritize communication structure (who calls whom, and what each output must contain) over role wording.
• Outer loop (weights updated): trajectories from the best workflows become supervised fine-tuning data. The orchestrator's initial workflow prompt is swapped for the bare task, so the student has to reproduce the team behavior without being handed the chart. Training uses a fresh LoRA adapter (rank 64, alpha 128, dropout 0.05, AdamW at a peak learning rate of 3e-5). The second cycle trains a new adapter on the merged first-cycle model.

Think of a restaurant kitchen that first rearranges who preps, who plates, and who hands off to whom (the workflow), then has the head chef study the tapes of the best service until next week's brigade runs that way without the laminated chart on the wall. The chef is also the critic grading the tapes, which is the part the paper has to work hardest to justify.

The setup is smaller than the word "recursive" suggests

MASS trains on twelve synthetic research tasks, four each in finance, robotics, and pharmacy. Nine are training tasks and three are held out. The first cycle's fine-tuning set is 119 trajectories, cut into 2,435 training windows totaling 17,221,055 supervised tokens. The supervised step alone took about 19 hours on six NVIDIA H100s, a figure that excludes workflow search, trajectory collection, and evaluation, which the paper does not total.

That is the useful part for anyone running a local rig: nothing here needs a cluster. The expensive piece is the search and the rollouts, not the gradient step.

Results on the paper's own tasks

On the three held-out synthetic tasks, the first-cycle model beats the base model in 53.9% of pairwise comparisons. The second-cycle model beats the base 69.9% of the time, and beats the first-cycle model 60.8% of the time. On the training tasks the numbers are 65.6% and 82.5%.

The evaluator improves too, which matters because the evaluator is the same model. Agreement between the in-loop judge and external judges rose from 73% at the base model to 91% after cycle one and 93% after cycle two. The authors argue this is what makes the loop "sustainable": a better model is also a better optimizer and a better critic for the next round.

Why multi-agent traces, not just more data

The paper's second claim is about data efficiency. A student fine-tuned on the multi-agent traces reached a 68.3% win rate using 28.8M supervised tokens. A single-agent student trained on 42M tokens reached 64.0%. That is 4.3 points higher on 31.4% fewer tokens. The catch is on the collection side: multi-agent episodes averaged 141,844 generated tokens versus 59,355 for single-agent runs, a 2.39x premium for producing the richer data in the first place. You pay for the teacher's talking.

Public benchmarks: research up, software engineering flat to down

The paper also reports six public benchmarks, each run three times per model with the official evaluators (five runs for one MLR-Bench configuration). Sakana's preferred metric is score per output token, which rewards doing the same work with less chatter. On that metric the second-cycle model reaches 1.60x the base on MLR-Bench, 1.35x on DSBench, 1.28x on AstaBench E2E-Bench-Hard, and 1.20x on ScienceAgentBench. On Terminal-Bench 2.0 it is 0.96x and on SWE-bench Verified 0.94x.

Raw scores from the repo's README tell the same story with less flattering framing. ScienceAgentBench goes 28.8% to 30.4% to 31.7% across the base, cycle one, and cycle two. MLR-Bench papers actually delivered rise from 57% to 76%, and AstaBench reports delivered from 50% to 76%. DSBench is essentially flat (0.473 to 0.482). Terminal-Bench 2.0 falls from 37.5% to 35.2%, and SWE-bench Verified from 62.6% to 61.1%. Several of those deltas sit inside one or two standard deviations of three-run noise.

The authors' explanation for the software-engineering dip is blunt: "We suspect this generalization can be easily addressed by introducing software engineering tasks into the training data." Plausible, and also untested in this paper.

Caveats the paper states itself

• Noisy search. The write-up says plainly that "workflow search is noisy," that the method "uses empirical workflow search and SFT without formal guarantees," and that the selected workflow is the best self-judged one, "rather than a certified global optimum."
• No monotonic promise. The loop "provides no guarantee that every future cycle improves." Two cycles were run. Whether cycle five still climbs is an open question the paper lists as future work.
• Self-judging. The search "does not isolate the optimizer from the other roles," so the same weights that propose a workflow also grade it. The 93% agreement with external judges is the mitigation, not a cure.
• MLR-Bench is aggregation-sensitive. The appendix admits the evidence there "depends on the aggregation unit, and does not establish a consistent gain across research tasks." The README's 107-task table and the paper's 12-task table also report different absolute scores.
• Partial release. The repo ships the pipeline, cycle configs, the twelve synthetic tasks, and adapters for ScienceAgentBench and a 12-task MLR-Bench subset. Adapters for AstaBench, DSBench, Terminal-Bench 2.0, and SWE-bench Verified are not included, weights are not bundled, and at the time of writing the file listing shows a NOTICE.md but no LICENSE file.

Why Sakana is pushing this now

Sakana launched a dedicated RSI Lab this autumn, with Jürgen Schmidhuber as Chief Scientific Advisor, and its pitch is sample efficiency over compute: redesign the development process with AI rather than buy more GPUs. The lab's prior exhibits include the Darwin Gödel Machine, which Sakana says more than doubled its SWE-bench baseline by rewriting its own agent code. MASS is the first paper from that lineage to tie workflow search to actual weight updates and to show the loop turning twice.

For builders, the reusable idea is narrower than "recursive self-improvement" and more interesting: your agent harness is training data. Spend compute evolving the orchestration, then distill the orchestration back into the model so the harness can be thrown away. On research-shaped tasks that bought a 20-60% per-token gain for a 27B model and a few days of H100 time. On coding tasks it bought nothing yet, which is a useful data point for anyone about to fine-tune a code agent on its own transcripts.

Key Takeaways

• MASS alternates evolutionary search over multi-agent workflows (weights frozen) with LoRA fine-tuning on the best team's transcripts (workflow text removed), using one Qwen3.6-27B model as executor, optimizer, and judge.
• Two cycles raised the held-out win rate against the base model to 69.9% and lifted judge agreement with external evaluators from 73% to 93%.
• Score per output token improved 1.20-1.60x on ScienceAgentBench, AstaBench, DSBench, and MLR-Bench, but came in at 0.96x on Terminal-Bench 2.0 and 0.94x on SWE-bench Verified.
• The multi-agent student beat a single-agent student by 4.3 points on 31.4% fewer supervised tokens, though collecting those traces cost 2.39x more generated tokens.
• Scale is modest: 119 trajectories, 17.2M supervised tokens, about 19 hours of SFT on six H100s per cycle, excluding search and evaluation.
• Code is public, but benchmark adapters are partial, weights are not bundled, and the paper gives no guarantee later cycles keep improving.

Sources: arXiv 2610.12176 (paper), Sakana AI write-up, SakanaAI/mass on GitHub, Sakana RSI Lab announcement, Hacker News discussion