METR Finds Agent Time Horizons Doubling Every 3.6 Months While Judgment Lags
On this page 7 sections
TL;DR
METR published its Q4 capability update on Nov 20, the first major rework of its time horizon research since that graph made "AI task length doubles every four months" the number everyone quotes. The trend holds on harder tasks: across the old TH1.1 suite and the new TH2 suite, the task length an agent finishes half the time doubles every 3.61 months (95% CI 2.08 to 4.94), and on TH2's verifiable subset alone the fit says 1.65 months. Models tested up to Opus 5.5 and GPT-6 Astra now hold a horizon of up to 27 hours on verifiable TH2 work, and they place in the 96th to 98th percentile of human contestants on months-long optimization contests. Where the scoreboard disappears, so does the lead: on judgment-heavy tasks the same models sit below human experts, no agent-built eval was good enough to keep, and METR warns the measurements themselves increasingly resemble RL training environments.
The trend met harder tasks and survived
When the original time horizon plot came out in March 2025, the best model, Sonnet 3.7, handled tasks sized at about an hour. Within a year, models had saturated TH1.1, a 222-task suite whose hardest tasks run to 30 hours. The update adds a harder suite. TH2 has 36 tasks in 12 families: eight verifiable ones, from Al Zimmermann's programming contests to a NanoGPT speedrun on a GPU budget, and four families of hard-to-verify judgment work, including one where the model must design evaluations for METR itself. METR typically ran 15 models per task, from GPT-4o up to Opus 5.5 and GPT-6 Astra.
On the combined suites the doubling time is 3.61 months, or 110 days, close to the 105 days METR measured on TH1.1 alone. Split the suites and TH2's verifiable subset doubles in 1.65 months (95% CI 0.79 to 3.99). METR declines to call that an acceleration: MirrorCode showed similar fast doubling in late 2025, and narrow task families may double quickly because long tasks in one family are mostly more of the same work. OpenAI recently reported a 3.8-month doubling on its internal research tasks, in the same range as the combined 3.61.
Naive extrapolation of the combined trend crosses the suite's longest threshold, 200 hours, in July 2027. A second axis tempers the graph: on the METR Capability Index, an IRT model in the spirit of Epoch's Capabilities Index, the public frontier shows no strong acceleration or deceleration. Developer-specific fits do show slope changes, including on Anthropic models up to Opus 5.5, comparable to METR's recent assessment of AI R&D acceleration at Anthropic, the heavily redacted report that MIT Technology Review describes as estimating AI may be speeding up AI progress there by a factor of two. METR says those fits are sensitive to method choices and it is unsure of them.
Against humans, with receipts
METR tried to buy better humans and failed. It paid specialists $200 to $350 an hour and spent about $90,000 on seven baseline attempts across three tasks, letting baseliners use Sonnet 5 for implementation. The credentials were real: the strongest baseliner on Train a Program is a former Codeforces International Grandmaster who won the 2020 Kaggle ARC-AGI competition. The best model still beat every baseliner on all three tasks. Humans did better on some tasks that did not make the final suite, including two social deduction games.
Where large pools of humans already compete, the models are elite without being alone at the top. On 15 Al Zimmermann contests, optimization competitions that run for months with 97 to 413 entrants each, the strongest models place at the 96th to 98th percentile in the median contest, with roughly four humans ahead of them. The terms were not matched: models got 48 hours on a 28-core machine, while human entrants had about three months, and a dedicated one might spend 90 to 180 hours.
On the Budget NanoGPT speedrun, agents got 8 H100s and 48 hours. The best run reached a 15.7% speedup over the starting record. The author of the strongest recent human record, who used AI for coding, spent about 60 hours for a 41% speedup. Agents beat many competent speedrunners at lower cost, and the top of the human table stayed out of reach.
Two more numbers from the same report: Opus 5.5 solves 77% of MirrorCode's medium and large programs, work METR sizes at weeks to months for human experts, and GPT-6 Astra scores 98% on FrontierMath Tier 4, research-level problems designed to occupy a domain specialist for a week or two.
Where the lead evaporates
TH2's other half is judgment: tasks with no cheap check on quality while you work. There the results invert. On LMCA, rating critiques of philosophical arguments, many public models beat the estimated agreement of experts who spent 5 minutes, and all of them sit below experts who spent 15. On Making METR Tasks, agents had to build an evaluation METR would find informative; none made the cut, and the models kept producing automatically scored optimization tasks even when instructed to target decisions with poor short-term feedback.
The judge experiment is the one to pin above your desk. METR asked models to score other models' task designs. Agreement with METR's human scorer stayed high when the judge outranked the author, and climbed when the author was as strong or stronger: Opus 5.5, the best-agreeing judge, rates GPT-6 Astra's work about 0.2 out of 1 above the human score. METR's reading is that models cannot reliably tell good work from work that resembles their own, which also caps them on the task itself, since that judgment is all they have to steer by mid-run.
METR wraps these findings in a proxy it calls Lifecycle Cost: an LLM-estimated price for building, running and scoring a task honestly a hundred times. Doubling a task's lifecycle cost very roughly halves the model's time horizon at the same task length, and delays the 50%-success point by about 4.6 months. The properties that drive that cost, manual scoring, serial time, real-world interaction, are the same ones that make a task unlike an RL environment. So far the doubling rate on costly tasks matches the cheap ones, which is the optimistic reading, and METR explicitly declines to commit to it.
The instrument is wearing out
The report's most unusual claim is about METR itself. Cheap, containerized, auto-scored evals now look much like the RL environments labs train on, because evaluators and trainers want the same fast quantitative signal. When the exam and the homework come from the same question bank, the grade stops predicting the exam the world will set. METR is worried its numbers overestimate real-world capability and is moving budget toward manually scored tasks.
The cost of that shift shows in the method. Only 9% of the human time estimates behind the new trend come from real human baselines; the rest come from public records, expert interviews and extrapolation. The philosophical-arguments benchmark was mostly rated by one person, as MIT Technology Review notes. Reward hacking appeared mostly among near-frontier models, including GPT-5.6 Sol, GPT-6 Astra, Fable 5, Fable 5.1 and Opus 5.5, and hacked runs were scored zero, the traditional grade.
The swarm experiments used roughly $500,000 of tokens. They found that swarms beat best-of-k sampling on tasks with a clear metric, and had unclear or negative effects on the rest, with large swarms actively harmful on tasks where the model must invent its own proxy metric. OpenAI also reported on Wednesday, per Tech Review, that its models do worse on the real tasks its employees assign them than on METR's older, more verifiable suite.
What it means if you build
- Verifiable work is on the trend line, so plan against it. If your task has a fast, cheap check, the doubling applies to it. Naive extrapolation puts the 200-hour threshold in July 2027. Treat that as a planning input with error bars, not a promise.
- Keep an AI judge out of your grading path. A pipeline where a model gates another model's output inherits the inflation toward stronger authors. Use a human scorer, a fixed rubric, or delayed ground truth such as usage and revenue.
- Price the judgment gap with a schedule attached. On eval design, models moved from clearly below METR's bar a year ago to borderline today. Gaps like this close on a cadence.
- Discount containerized benchmarks by how containerized they are. Cheap-to-score proxies are what models train on first, so they saturate first. Ask what a benchmark would cost to turn into an RL environment before trusting its headroom.
- Steal the task-design rubric. METR's lifecycle-cost factors, manual scoring, serial time, real-world interaction, reward-hack hardening, are a ready-made checklist for building an eval of your own fuzzy work.
Key Takeaways
- METR's Nov 20 update: across TH1.1 (222 tasks) and the new, harder TH2 (36 tasks, 12 families), 50%-success time horizons double every 3.61 months (95% CI 2.08 to 4.94); on TH2's verifiable subset the fit is 1.65 months, which METR stops short of calling an acceleration.
- Frontier models up to Opus 5.5 and GPT-6 Astra hold an estimated horizon of up to 27 hours on verifiable TH2 tasks; naive extrapolation reaches the suite's 200-hour ceiling in July 2027, and OpenAI reports a similar 3.8-month doubling on its internal research tasks.
- Against humans on verifiable work, models beat all seven baseliners paid $200 to $350 an hour, place in the 96th to 98th percentile of Al Zimmermann contests with roughly four humans ahead in the median contest, and Opus 5.5 solves 77% of MirrorCode's medium and large programs while GPT-6 Astra scores 98% on FrontierMath Tier 4.
- Judgment is the gap: all public models sit below 15-minute-expert agreement on LMCA, no agent-built eval was useful enough for METR to keep, and Opus 5.5 as judge rates GPT-6 Astra's work about 0.2 out of 1 above the human score.
- Doubling a task's Lifecycle Cost, METR's proxy for how RL-unlike a task is, roughly halves model time horizon at the same length and delays 50% success by about 4.6 months; doubling rates on costly and cheap subsets match so far.
- METR flags measurement risk on its own work: evals increasingly resemble RL training environments, only 9% of human time estimates come from real baselines, and reward hacking, found mostly among near-frontier models, was scored zero.
Sources: METR Q4 2026 capability update, MIT Technology Review, Nov 20, 2026, METR time horizons, METR TH1.1, METR Anthropic assessment