RTK Says It Saved 349M Tokens. The Bill Went Up 17%.
TL;DR
RTK, the Rust CLI proxy that compresses terminal output before your coding agent reads it, carries 80,010 GitHub stars and a repo tagline promising to reduce "LLM token consumption by 60-90% on common dev commands." On September 11, Quesma published the second independent test of that claim with real money on the line: 1,740 agent attempts on Terminal-Bench 2.1, more than $1,500 of tokens burned. RTK's own counter reported 349.2 million tokens saved across the DeepSeek arm. Average per-task cost in that same arm went up 17%. On Claude it moved 1%, which is to say it did not move.
The authors' conclusion, verbatim: "We do not recommend RTK as a generic cost-saving tool."
What RTK actually does
RTK installs a hook that rewrites the shell commands your agent runs. Instead of ls -la warriors/ returning owner, group and timestamp for every file, RTK hands the model 644 paper.red 487B and drops the rest. It does the same for git diff, git log, pytest, cargo test, docker ps and about a hundred other commands: failures only, tracebacks trimmed, passing tests collapsed to a count.
This is a genuinely good idea, executed well. It is a single Rust binary under Apache 2.0, installs from Homebrew, adds under 10ms of overhead, and works with Claude Code, OpenCode, Cursor, Aider and anything else that reads a terminal. Eighty thousand people did not star it by accident.
The problem is the arithmetic between "less output" and "smaller invoice."
The test
Quesma ran RTK 0.45.0 against Terminal-Bench 2.1, an 89-task suite maintained by Stanford, Harbor and the Laude Institute that is deliberately heavy on terminal interaction. Two harness and model pairs: Claude Code 2.1.220 with Fable 5.0, and OpenCode 1.18.25 with DeepSeek V4 Pro 0813 routed through OpenRouter. Every task was scheduled five times with RTK and five times without, same route, same platform, same timeout. After dropping four Fable tasks that drew safety refusals, the comparison covers 85 Fable tasks and 89 DeepSeek tasks, 1,740 attempts total.
The totals looked fine. One task was doing all the work.
At the aggregate level the numbers are unremarkable. Fable spent $731 baseline and $698 with RTK, a 5% drop, with pass rate slipping from 84% to 83%. DeepSeek spent $51 baseline and $54 with RTK, a 5% rise, pass rate 71% to 69%.
Then they weighted every task equally, which is the honest way to do it when one expensive task can swamp a hundred cheap ones. Fable came out 1% more expensive with a confidence interval straddling zero. DeepSeek came out 17% more expensive. Restricting to the 36 DeepSeek tasks where all ten attempts passed, so failures cannot be blamed, the increase was still 18%.
Almost all of Fable's headline 5% saving traced to a single task, winning-avg-corewars, where the RTK run happened to finish in about half as many turns. Strip that one task out and the savings across the other 84 fall below 1%. DeepSeek ran the same task and got the opposite result: more turns, more money.
Why the compression never reaches your invoice
Three things dilute it, and they multiply.
Terminal output is a small slice of input. Without RTK, terminal output was about 7% of Fable's input tokens and 26% of DeepSeek's. All tool output together, terminal plus everything else, came to 11% and 40%.
RTK only sees the Bash tool. Claude Code and OpenCode expose file reading and searching as separate Read, Grep and Glob tools, which route around the hook entirely. In practice only 31% of Claude Code's terminal calls and 51% of OpenCode's went through RTK. Roughly half of Claude Code's Bash calls were already capping their own output with head, tail or wc, because frontier models learned to do that on their own.
Most input tokens are cache reads, and they are cheap. Agentic context is cached after each turn, so re-reading that terminal output later bills at one tenth the normal input rate for Fable and one thirtieth for DeepSeek. Cache reads were 94% of Fable's input tokens but only 30% of the bill. Model output, including reasoning, accounted for 56% of DeepSeek's cost.
Put it together and you are haggling hard over the parsley on a bill that is mostly steak.
The turn tax
Here is where it goes from "no help" to actively negative. Compressed output means the model sometimes has to go back and ask again.
The average DeepSeek turn carried 7% less input with RTK. There were 18% more turns. Prompt tokens rose 9% overall, entirely on the cached side. RTK attempts took more turns on 58 tasks, and 44 of those cost more; fewer turns on 28 tasks, and 23 of those cost less. The direction is consistent.
The counter measures bytes, not money
RTK ships a gain command, and this is where the viral 90% numbers come from. It computes raw output minus filtered output in bytes, divided by four. RTK ships no tokenizer, so those are estimates of characters removed, not tokens billed.
Across 445 DeepSeek RTK attempts, gain reported 349.2 million tokens saved, an 89% reduction. Two of those calls did most of the reporting. On the train-fasttext task the model ran head -1 train.txt twice, and RTK credited itself 120.5 million tokens each time by diffing a one-line read against the entire file. Those two calls alone were 69% of the savings counter, measured against output that head -1 was never going to return in the first place.
The counter is not lying. It is answering a different question than the one on your invoice, and answering it loudly.
To RTK's credit, the README says so plainly: "RTK cuts up to 90% of the bash output your agent reads. That is what RTK measures, and it is not the same as cutting your bill by 90%." The repo tagline still says 60-90% token consumption reduction. An issue asking them to fix it has been open since July 22.
One bug, 339 errors, nine times the price
On a DeepSeek git-multibranch attempt, the agent called find with a flag that RTK 0.45.0 did not support. The hook rewrote it to rtk find, which failed with "Use find directly." Every retry got rewritten again. The agent accumulated 339 consecutive errors over roughly twelve minutes before timing out. It still passed the task, at about nine times the cost of the matching baseline attempt that also passed. RTK fixed this in 0.46.0, after the runs finished.
One outlier, and the authors note the trend holds without it. But a wrapper that can silently loop your agent into a wall is a different risk profile than a wrapper that merely fails to help.
What RTK says
This is the second independent benchmark to land on the same answer. JetBrains tested RTK in July on SkillsBench with 425 paired runs on claude-sonnet-5 and measured a median +7.6% per task at low reasoning effort (p=0.004) and +0.1% at high effort (p=0.99). Task quality was unchanged. RTK's counter claimed 96.2 million tokens saved in the same trials.
RTK responded on August 7, and the response is reasonable: SkillsBench is not a Bash-heavy suite, only 16 of its 86 tasks are software engineering, and agent variance is wide enough that single-trial results are noise. On their own 13-task developer sweep at four repetitions they measured -4.8%, with p=0.305, which is to say not distinguishable from zero either. They committed to expanding past Bash compression.
Terminal-Bench 2.1 is precisely the terminal-heavy, multi-repetition benchmark that objection asked for. It returned the same verdict.
What to do with this
Do not uninstall RTK because a blog post said so; measure your own bill. But update your prior. The tool was built when models padded their context with everything the shell handed them. Current frontier models already reach for head -n and tail -n without being asked, which is exactly why terminal output is down to 7% of Claude's context. RTK is solving a problem the models partly solved themselves.
The broader lesson is cheap and reusable: any optimization measured in "stuff removed" needs a second number measured in dollars, on your workload, against a control arm. Token counters report a counterfactual that never ran. Your invoice reports what happened.
Key Takeaways
- Two independent benchmarks, same answer. Terminal-Bench 2.1 at 1,740 attempts and SkillsBench at 425 paired runs both found RTK costs more or the same, never less.
- Per-task cost rose 17% on DeepSeek V4 Pro and 1% on Fable 5.0, with the Fable interval crossing zero. Pass rates dipped 1 to 2 points.
- The 349.2 million "tokens saved" is a byte counter. Two
head -1calls diffed against a whole file produced 69% of it. - Terminal output is 7% of Claude's input tokens, most input is cached at a tenth the price, and the hook only catches 31% of terminal calls. The savings dilute at every step.
- Compression buys turns. 7% less input per turn, 18% more turns, 9% more prompt tokens overall.
- Measure the invoice, not the counter. Any "we removed X%" claim needs a controlled cost comparison on your own workload before you believe it.
Sources: Quesma, "RTK reports huge token savings, but our cost benchmarks disagree", JetBrains, "rtk Claude Code Token Savings: A Skill Trial Benchmark", rtk-ai/rtk on GitHub, RTK, "RTK on SkillsBench: What the Benchmark Measures", rtk-ai/rtk issue #3157, Terminal-Bench 2.1 release notes