METR Put a Price Tag on Autonomous AI R&D. The Best Agent Is Worth $3,300 Before a Human Beats It.

TL;DR

On July 21, METR published Expenditure Horizon, a new way to score AI agents that throws out pass/fail and returns a dollar amount instead: how much you can spend driving an agent at an optimization problem before a human with the same budget would have done better. Measured against the NanoGPT speedrun, the best model tested lands around $3,300. Two of the six models score exactly zero. For context, humans have poured roughly $250,000 of labor into the same problem.

What an expenditure horizon actually is

METR's definition is blunt: an agent's expenditure horizon on an optimization problem is "the dollar value at which the improvement to the goal metric is equal to the improvement by a human with the same budget."

Think of it as the crossover point on the receipt. Below it, buying an improvement from the agent is the cheaper move. Above it, you should have hired the person. It is the same logic as renting versus buying: the interesting number is not which is better, it is the exact spend where the answer flips.

Computing it needs two curves. One is the agent scaling curve, which is performance as a function of total spend on tokens plus experiment compute. The other is the marginal return to human labor on the same problem. Where they intersect is the horizon.

The testbed: a speedrun with a public receipt

METR picked the NanoGPT speedrun, an open competition to train a language model to a target validation loss of 3.28 cross-entropy on a held-out FineWeb sample using 8 NVIDIA H100 GPUs, as fast as possible. It started from Andrej Karpathy's 45-minute GPT-2 reproduction and has been ground down for two years by a public queue of contributors.

That public history is the whole reason the problem works as a testbed. Every record has an author, a diff, and a date, so METR could estimate the human side rather than guess at it. Through contributor interviews and LLM-assisted analysis, they put a 1% improvement at roughly 16 hours of labor, or $2,400 at $150 an hour, rounded to $2,500 per 1% improvement. That hourly rate stands in for about a $300K annual junior researcher.

Agents were dropped in at record #78 from March 2026, with a baseline training time of 85.56 seconds.

The numbers

GPT-5 and Opus-4.1 produced, in METR's words, "no or few optimizations." Their horizon is zero, which is a polite way of saying they spent your money and returned statistical noise.

The newer three behave differently. GPT-5.2, GPT-5.5, and Opus-4.8 keep improving roughly log-linearly with spend into the thousands of dollars, which is the actual finding here: the curve did not flatten early. Opus-4.8's $3,300 horizon corresponds to an estimated 1.5% speedup in the portion of its work that was actually mergeable. GPT-5.5's $2,300 maps to about 1%.

Push past $10,000 of spend on a single run and METR still calls the big-picture improvements modest.

Why $3,300 is smaller than it sounds

Here is the number that reframes the whole thing. From May 2024 to April 2026, the human contributors to the speedrun put in roughly 1,650 hours, or about $250,000 at METR's assumed rate.

METR's conclusion follows directly: autonomous agent optimization has so far had minimal effect on AI R&D progress in NanoGPT. If you have been reading "AI is now doing AI research" as a description of the present rather than a forecast, that sliver is the correction.

Why this beats the stopwatch

METR is the group behind the widely cited time horizon metric, the one that reports how long a task a model can complete with 50% reliability. Expenditure horizon is an explicit patch on two things that metric gets wrong.

First, binary pass/fail throws away information. A task you 90% finished and a task you never started both score zero. Continuous scoring on an optimization problem extracts far more signal per expensive agent run.

Second, wall-clock time does not specify a budget. Two agents can both "take an hour" while one burned 40x the tokens and 10x the GPU-hours. Denominating agent tokens, experiment compute, and human labor all in dollars makes the comparison honest.

It also rhymes with METR's earlier RE-Bench result, where agents beat human experts at ML research engineering given 2 hours and lost badly at 32. Agents sprint. Humans have longer legs. Expenditure horizon is the dollar-denominated version of the same shape, and it applies cleanly to MLE-bench-style suites too.

The caveats METR puts on its own number

Read these before you quote $3,300 at anyone. METR lists eight limitations, and several are load-bearing.

• It measures autonomy only. This is the big one. A human using an agent is not measured here, and that is the configuration almost everyone actually ships. The number is a floor on usefulness, not a ceiling.
• Contamination cuts both ways. If a lab already trained against the problem, the measured horizon comes out unrepresentatively short. If a model's knowledge cutoff lands after some optimization was published, it comes out inflated.
• Lumpy returns break the model. Domains where progress arrives as rare breakthroughs rather than steady grinding do not fit a smooth intersection.
• Noise nearly ate the result. NanoGPT's timings are jittery enough that METR had to revalidate extensively, and attributes some overstatement to statistical noise.
• The framework has an expiry date. The whole construction assumes returns to agent spend diminish faster than returns to human spend. METR says plainly that it doubts this stability holds over a wider range. Once it fails, the intersection stops existing and the metric is undefined.

That last bullet is the quiet one. The metric is built to describe a world where humans still out-scale agents at the top end, and it explicitly tells you it will stop working when that changes. A ruler that announces its own obsolescence date is unusual, and more useful than one that does not.

What to do with this if you build agents

The practical takeaway is a budgeting discipline, not a leaderboard position. If you are pointing an agent at a real optimization problem, the question is not "can it do it," it is "at what spend does it stop being the cheap option." That number is knowable, it differs by more than 3x across one model generation, and for two of the six models tested it was zero.

Also note what METR did not need: a private eval, a vendor's cooperation, or a held-out test set. It needed a public problem with a public commit history and honest accounting on both sides. There are more of those lying around than the industry uses.

Key Takeaways

• METR's expenditure horizon is the dollar spend at which an agent's improvement equals a human's improvement on the same budget, replacing pass/fail with a continuous, cost-denominated score.
• On the NanoGPT speedrun, horizons ranged from $0 to $3,300: GPT-5 and Opus-4.1 at $0, GPT-5.2 near $600, GPT-5.5 near $2,300, Opus-4.8 near $3,300.
• The human baseline is roughly $2,500 per 1% speedup, derived from about 16 hours of labor at $150 an hour.
• Humans have spent about 1,650 hours and $250,000 on the problem since May 2024, so the best agent horizon is a rounding error against it. METR concludes autonomous agents have had minimal effect on this line of AI R&D.
• Newer models keep improving log-linearly with spend into the thousands of dollars, which is the real generational signal. Past $10,000 per run, gains are still modest.
• The metric measures autonomous work only, so it likely understates human-plus-agent workflows, and METR says the framework becomes undefined if agent returns ever stop diminishing faster than human returns.

Sources: METR, Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT, modded-nanogpt (NanoGPT speedrun), METR, Measuring AI Ability to Complete Long Tasks, METR, RE-Bench