Same Model, 12 Harnesses, and a 17x Spread on the Bill

TL;DR

FrontierHarness Eval, published September 1 by agent-runtime vendor Runta, holds the model constant and varies the harness: Kimi K3 served through Fireworks, 30 tasks, 12 configurations of 9 harnesses, 360 runs, every one a fresh restore of the same frozen checkpoint. Pass rates came in between 50.0% and 66.7%. Cost per solved task came in between $1.05 and $18.34. The results JSON is public, and doing the arithmetic on it yourself turns up the number the leaderboard does not print: of the $1,000.46 the whole eval burned, $745.51 went to tasks that failed.

The setup, which is the interesting part

Harness benchmarks barely exist because they are annoying to run fairly. Every harness ships its own system prompt, its own tool surface, its own context management, and its own idea of when to stop. Swap the model too and you cannot attribute anything.

Runta pinned everything else down. One model. One inference provider. Thirty tasks: 21 from Terminal-Bench and 9 repo-level bug fixes from datacurve against real projects (fastapi, httpx, katex, arktype, meriyah, python-statemachine). Every run starts from a golden checkpoint restored down to memory state, with identical vCPU, RAM and disk contents, so no harness gets a warm cache the others do not. Harness versions are pinned in the repo metadata: Codex 0.148.0, Claude Code 2.1.237, OpenCode 1.18.19, Kimi Code 0.37.2, Pi 0.84.2, Oh My Pi 17.4.0, Hermes 0.20.4, Exo 0.1.0, and four DeepSeek Harness profiles at 0.1.0-rc.8.

Zero runs were thrown out as infrastructure failures. That is a cleaner control than most model benchmarks manage.

The 17x is real, and it is mostly cache

Claude Code posted the joint second-best pass rate at 63.3%, 19 of 30, one task behind Codex. It also spent $348.40 doing it, just over a third of the entire eval's spend, on someone else's model.

The mechanism is in the cache columns. Runta reports two cache hit rates, and the gap between them is the whole story. By median run, Claude Code hit cache 67.8% of the time, which looks unremarkable. Token-weighted, across the runs that actually consumed the tokens, it hit 24.9%. Every other harness sits between 91.6% and 99.1% token-weighted.

Prompt caching is a print shop that keeps your book's plates set. Send the same opening pages and they reuse the setup and charge you a fraction. Change something on page two and they reset all four hundred pages at full price, and they do it again on every single turn. A harness that quietly rewrites the front of its context each turn is paying full freight for a context that grows all day.

Look at the individual runs and it gets concrete. Claude Code's most expensive single trial was a datacurve task called meriyah-explicit-resource-declarations: 306 turns, a flat 60 minutes of wall clock, 19.2% cache hit, $77.64, and it failed. Exo's entire 30-task run, successes and failures together, cost $16.72.

The caveat that matters here: this is Claude Code driving Kimi K3 through a third-party endpoint, not Claude Code driving Anthropic models. Harnesses tuned around one provider's caching semantics do not automatically get them elsewhere, and Runta's own author conceded the point in the Hacker News thread, calling it a known v1.0 limitation. Read the copper bar as a statement about harness-provider fit, not about Claude Code on its home turf.

Three quarters of the money bought nothing

This part is not on the leaderboard. Sum the per-task costs in the results file and split them by outcome: the eval spent $1,000.46 in total, $254.95 on the 209 runs that passed and $745.51 on the 151 that failed. Nearly three dollars in four bought a task that was never finished.

OpenCode is the sharpest illustration. It is the most frugal harness in the set on paper: 153,533 mean input tokens per task and 11.2 mean turns, an order of magnitude below Codex's 4.7 million tokens and 62.4 turns. Yet 96.4% of its $48.67 went to failures, because when OpenCode does not converge it does not give up either. Its four costliest runs ran 350, 351, 234 and 197 turns, and all four failed.

That is the pattern worth internalizing: your agent bill is not set by how efficient the happy path is. It is set by what the harness does when it is lost.

The scoreboard is thinner than it looks

Sixteen and a half points separate the best pass rate from the worst, which sounds like a real ranking. It is five tasks out of thirty, and most of the benchmark carries no signal at all.

Twelve tasks were solved by all twelve configurations. Three were solved by none. The entire ranking rests on the fifteen in the middle. The split by source is stark too: the Terminal-Bench half passed 183 of 252 runs (72.6%), the datacurve half passed 26 of 108 (24.1%). Nine repo-level bug fixes are carrying the discrimination and most of the spend.

Then there is sample size. One run per task-harness cell, no repeats. At a 60% pass rate on 30 tasks the binomial standard error is about 8.9 points, so a 95% interval spans roughly 35 points. Every harness in this eval sits inside every other harness's interval. A commenter on Hacker News raised exactly this, and the arithmetic backs them: treat the pass-rate column as noise and the cost column as signal.

What to do with this on Monday

• Measure cache hit rate token-weighted, not per-run. Claude Code's 67.8% median and 24.9% weighted describe the same runs. Only one of them predicts the invoice.
• Cap turns and wall clock. Exo never exceeded 51 turns on any task in the set. Its whole 30-task run cost less than a quarter of one uncapped hour-long failure elsewhere.
• Budget for failures explicitly. If three quarters of your agent spend is going to unfinished work, cost-per-success is your only honest unit, and a cheap failure loop beats an efficient happy path.
• Do not switch harnesses on a 30-task pass rate. Switch on cost per success, cache behavior, and how the thing behaves when it is stuck.
• Check harness and provider fit before you blame the model. Most of the spread here is a harness meeting an endpoint it was not tuned for.

Where to be skeptical

Runta sells the runtime the eval ran on and will hand you $100 in credits to bring your own harness, so this is a vendor benchmark, however clean the controls are. The repo publishes results, task definitions and the evaluation workflow but not the adapters or infrastructure, which means you can audit the numbers and cannot independently reproduce them. And the DeepSeek Harness occupies four of the twelve configuration slots, which is a lot of shelf space for one vendor's profiles.

The harness set is also uneven by design. Pi is deliberately minimal, and Oh My Pi is essentially Pi plus a default extension bundle. Here the bundle scored lower (56.7% versus 60.0%) and cost nearly twice as much per solve ($4.75 versus $2.43), which is either a useful finding about extension overhead or an artifact of a single run each. On thirty tasks with no repeats, you do not get to know which.

Runta says v1.1 moves to a full harness by model matrix. That is the eval everyone actually wants, and it is the only version that can separate "this harness is wasteful" from "this harness has never met your provider."

Key Takeaways

• Holding the model fixed, harness choice moved cost per solved task 17.5x ($1.05 to $18.34) while moving pass rate only 16.7 points, or five tasks out of thirty.
• Across all 360 runs the eval spent $1,000.46, of which $745.51 (74.5%) went to tasks that failed. Per harness the failure share ran from 48.7% to 96.4%.
• The cost outlier tracks cache, not capability: 24.9% token-weighted cache hit for Claude Code against 91.6% to 99.1% for everyone else, driving one 306-turn, 60-minute, $77.64 failure.
• Half the task set is dead weight: 12 tasks every config solved, 3 none did, and the ranking rests on the other 15 with one run apiece.
• Frugal on the happy path does not mean cheap. OpenCode used 153,533 mean input tokens and 11.2 turns yet burned 96.4% of its budget on runs that spiralled past 300 turns.
• It is a vendor benchmark with public results and private infrastructure. Use the cost findings, discount the leaderboard order.

Sources: FrontierHarness Eval, Runta blog: Introducing FrontierHarness Eval, frontier-harness-eval/eval results and methodology, Hacker News discussion, Terminal-Bench, Kimi K3 model card