Glonce

RTK doesn't cut AI coding costs, Terminal-Bench 2.1 finds

toolsThe Glonce editorsHacker News🔥68Glonce rating68Impact60Novelty55Relevance95Surprise65Players65Credibility88
RTK doesn't cut AI coding costs, Terminal-Bench 2.1 finds

RTK, short for Rust Token Killer, filters and compresses the terminal output an AI coding agent reads before it goes back into the model's context. The tool has more than 79,000 GitHub stars, and one post on X claimed it could cut Claude Code's token usage by up to 60%, a claim that reached 313,000 views. RTK's own README carries a disclaimer that cuts against that framing: the tool can strip up to 90% of the bash output an agent reads, but that is not the same as cutting the bill by 90%. A separate benchmark run by JetBrains, called SkillsBench, had already found no savings from using RTK at all.

To test the claim directly, Quesma ran RTK through Terminal-Bench 2.1, a benchmark built around heavy terminal interaction. The team picked the 2.1 version over the newer 3.0 and 4.0 releases because agents already pass most 2.1 tasks, while the newer versions remain a challenge, and cost only matters for tasks that actually pass. Two setups were compared: Claude Code running the Fable 5.0 model, and OpenCode running DeepSeek V4 Pro 0813 through OpenRouter. Every task ran five times without RTK and five times with it, on the same model route, platform and task-specific timeout. After dropping four Fable security tasks that produced refusals, the final comparison covered 85 Fable tasks and 89 DeepSeek tasks, a total of 1,740 attempts, and consumed over $1,500 in tokens across several days of testing.

The headline numbers were mixed from the start. At the raw aggregate level, RTK cut Fable's costs by 5% and raised DeepSeek's by 5%; pass rates dipped slightly with RTK for both models, by 1% for Fable and 2% for DeepSeek, gaps the team calls small. Dividing total spending, including failed attempts, by the number of passes told a similar story: Fable came out 3% cheaper with RTK, DeepSeek 7% more expensive. Weighting every task equally instead, so one expensive task cannot dominate the average, Fable showed a 1% cost increase with no clear difference from zero, while DeepSeek's average task cost rose 17%. Restricting the comparison to the 36 DeepSeek tasks where all ten attempts, five baseline and five with RTK, passed did not change the picture: the increase there was still 18%.

Almost all of Fable's apparent savings traced back to a single task, winning-avg-corewars: both setups passed every attempt, but with RTK the run finished in about half as many turns. Across every other task, the saving fell under 1%. DeepSeek showed the opposite pattern on that same task, again passing every attempt in both setups, but taking more turns and costing more with RTK; even with that task excluded, DeepSeek's costs stayed higher across the board.

The team also examined RTK's own headline metric, rtk gain, and found it measures something else entirely: raw command output minus filtered output, in bytes, divided by four, not a count of tokens actually billed. Across 445 DeepSeek attempts run with RTK, the tool reported 349.2 million tokens saved, an 89% reduction by its own count, but that number did not translate into cheaper runs. In one task, train-fasttext, the agent ran head -1 train.txt twice; RTK credited 120.5 million tokens saved for each call by comparing that limited read against the size of the entire file, even though the command itself would never have returned the whole file. Those two calls alone accounted for 69% of the entire comparison's counted savings. The team frames this as the core problem with how these tools get marketed: a metric that counts removed output is being read as money saved, when treating it that way assumes the rest of the attempt stays the same, and it can make a run that actually costs more look optimized.

Testing also surfaced a live bug. In one DeepSeek attempt on a task called git-multibranch, the agent ran a find command with a flag that RTK 0.45.0, the version tested, did not support. RTK's plugin rewrote the call to a plain rtk find, which failed with the error 'Use find directly'; every retry was rewritten the same way and failed the same way, and the agent accumulated 339 consecutive errors before timing out. The task still passed, but that single attempt cost about 9x as much as its matching baseline run, which also passed. RTK fixed the underlying issue in version 0.46.0, released after these tests were run.

The deeper problem, per the team, is that terminal output was never a large share of the bill to begin with. Without RTK, tool output made up about 11% of Fable's input tokens and 40% of DeepSeek's. Even in the RTK-enabled runs, RTK touched only a minority of terminal calls, 31% for Claude Code and 51% for OpenCode, partly because RTK rewrites only the shell tool, Bash in Claude Code and bash in OpenCode, and not the separate Read, Grep and Glob tools both platforms already use for file access and search, and partly because about half of Claude Code's Bash calls already limited their own output with head, tail or wc before RTK ever saw them. Caching narrows the potential gain further: because agentic coding caches context turn to turn, a cache read costs a tenth of a regular input token for Fable and a thirtieth for DeepSeek, so trimming output that would mostly be served from cache anyway buys less than it appears to. In DeepSeek's case, RTK cut terminal-output characters by 9%, yet prompt tokens still rose 9%: uncached input fell 1% while cached input rose 9%. Model output, including reasoning, made up 56% of DeepSeek's cost with RTK and 57% without it.

Turns were the recurring driver of cost. On the 58 DeepSeek tasks where RTK led to more turns than the baseline, 44 also cost more; on the 28 tasks where RTK led to fewer turns, only 23 cost less. On average, each RTK-assisted DeepSeek turn used 7% less input, but there were 18% more turns overall, so the smaller turns did not add up to less total input. JetBrains's SkillsBench benchmark saw the same shape: RTK added turns at low effort settings and did not lower cost at high effort.

The team's stated conclusion: RTK does not make AI coding cheaper. On Terminal-Bench 2.1, Fable's savings depended on a single task and did not hold up across the rest, so the team does not recommend RTK as a generic cost-saving tool. Individual transcripts showed that current frontier models already use the terminal efficiently on their own (just about 7% of Fable's context was terminal output), and models already reach for head -n or tail -n themselves; the team believes RTK probably helped more against older models and is now a niche optimization rather than a general one. The tests ran RTK 0.45.0, Claude Code 2.1.220, OpenCode 1.18.25 and Harbor 0.20, with trajectories available on request for follow-up research; a future post is planned to benchmark a different tool, Headroom, which this study did not itself test. The post does not name its own author; it credits only a reviewer, Piotr Migdał, for feedback, not authorship.

Key facts

  • RTK, a terminal-output compressor with over 79,000 GitHub stars, is credited by a viral X post with 313,000 views for cutting Claude Code's tokens by up to 60%; RTK's own README says its up-to-90% output cut is not the same as cutting the bill by 90%.
  • Quesma tested RTK across 1,740 agent attempts on Terminal-Bench 2.1, running Claude Code with the Fable 5.0 model and OpenCode with DeepSeek V4 Pro 0813, and spent over $1,500 on tokens to do it.
  • On a cost-per-completed-task basis, RTK made Fable 3% cheaper but DeepSeek 7% more expensive; weighting every task equally instead, Fable showed no clear change while DeepSeek's average task cost rose 17%, holding at 18% even within the 36 tasks where every attempt passed.
  • Nearly all of Fable's savings came from a single task, winning-avg-corewars, where RTK finished in about half as many turns; across every other task combined, the saving was under 1%, and DeepSeek got the opposite result on that same task.
  • RTK's own rtk gain metric measures raw output bytes removed divided by four, not billed tokens: across 445 DeepSeek attempts it reported 349.2 million tokens, an 89% reduction, but two repeated head -1 train.txt calls alone, each credited with 120.5 million tokens saved, made up 69% of that entire tally.

Why it matters

Viral claims about AI coding costs spread faster than anyone checks them. RTK has more than 79,000 GitHub stars and a widely shared claim, from an X post with 313,000 views, that it can cut Claude Code's tokens by up to 60%. That framing shapes how teams evaluate cost-saving tools for agentic coding, but this benchmark and a separate JetBrains test both found the number does not hold up, and that RTK's own gain metric can make a more expensive run look cheaper than it is. The gap between a tool's self-reported savings and what actually gets billed matters for anyone choosing infrastructure for AI coding agents on the strength of a headline number.

Who it affects

Anyone running Claude Code or OpenCode at scale and paying per token, since the study shows RTK's effect depends on which model sits behind the agent: cost fell for one, rose for the other. It also affects RTK's own maintainers, whose 0.45.0 release had a retry bug serious enough to run one attempt to 339 consecutive errors and a 9x cost spike before a fix shipped in 0.46.0. And it affects JetBrains, whose own SkillsBench benchmark reached a similar conclusion independently.

How to use it

The post is a critique, not a product pitch, but its method doubles as a checklist for testing any cost-saving coding tool: measure pass rate alongside cost, since a tool that fails more often is not actually cheaper; compute cost per completed task rather than trusting a built-in savings counter; and weight tasks equally so one outlier does not carry the whole average, since a single task, winning-avg-corewars, produced nearly all of the one positive result here. Versions matter too: this test used RTK 0.45.0, Claude Code 2.1.220, OpenCode 1.18.25 and Harbor 0.20, and the retry bug described here was already fixed in RTK 0.46.0, so results on the current release could differ.

How solid is it

This is a controlled, third-party test, not RTK's own numbers: 1,740 attempts across 85 Fable tasks and 89 DeepSeek tasks, five runs per arm per task on matched models, platforms and timeouts, for a stated spend of over $1,500. It is corroborated by an independent source: JetBrains's SkillsBench benchmark reported the same shape of result, more turns without lower cost. The caveats sit in the source's own numbers: no p-value or confidence interval is reported anywhere, so the small pass-rate gaps and Fable's 'no clear difference from zero' result are the team's qualitative read, not a formal significance test. A sample of 85 to 89 tasks per model is also small enough that a few unusual runs can move the average by a visible amount.

Risks and caveats

RTK's own gain metric is not a cost figure: it counts bytes removed divided by four, and in one task alone, two repeated head -1 train.txt calls generated 120.5 million tokens of credited savings each, 69% of the entire tally, for a command that never would have read the whole file. A live bug in the tested version rewrote unsupported retries into a call that failed every time, running one attempt to 339 consecutive errors and about 9x the cost of its baseline before RTK fixed it in 0.46.0. RTK also only touches shell commands, so file reads and searches through Read, Grep or Glob bypass it entirely, and even within shell calls it acted on just 31% to 51% of terminal calls in these tests. Results were model-specific too: whatever helped Fable made DeepSeek more expensive, so a result measured on one model and agent platform should not be assumed to transfer to another.

“This is where social posts go wrong: rtk gain counts removed output, not money saved, and it can make a more expensive attempt look optimized.”

— Quesma's writeup of the RTK benchmark