erabot.ai
ProductPricingCase studiesEvalBlogDocs
Book a demo

LLM token compression: what the 90% savings claims actually measure

TL;DR: LLM token compression works, but the headline numbers are measured on the easiest possible input. When you separate the claims by what was compressed, the 85–97% figures are almost always JSON, logs, shell output or tool manifests. For actual source code, the credible published figures cluster nearer 20–55% — and the vendors' own documentation says so if you read past the landing page.

The claims, sorted by what was actually compressed

We went through the current crop of compression tools and pulled the fine print behind each headline. The pattern is consistent enough to be useful:

  • Headroom advertises 60–95%, but its own README pegs savings for coding agents at roughly 15–20%. The high end is reserved for JSON.
  • rtk reports 60–90% — explicitly "on common dev commands." That is the output of git status and friends, not your repository.
  • Atlassian's MCP compressor reports 70–97%, and the striking example is real: a 94-tool MCP server going from about 17,600 tokens of tool definitions down to roughly 500. That is tool-schema overhead, a genuinely large and genuinely separate problem from compressing code.
  • Paritok quotes 85–95% for context-saturated multi-turn sessions. Its actual benchmark line is more interesting: context compressed to about 25.7% of original size while retaining 86.5% of solve quality on SWE-bench Lite — roughly a six-point drop in problems solved.

None of these are dishonest. They are just answers to a different question than "how much will this cut my code context."

Where the code numbers land

Restrict the question to source code and the spread narrows considerably:

  • LaMR holds top answer quality at −20.5% tokens.
  • SWE-Pruner reports 23–54% reduction.
  • Headroom's own coding-agent figure, again: ~15–20%.

That is the 20–55% band. It is not a law of nature — it is where several independent, published attempts happen to land once the input is code rather than logs. Treat it as "credible sources converge here," not as a measured constant.

One more result deserves its own line, because it cuts against the whole premise. A pre-registered production trial found that aggressive compression actually raised mean total cost by 1.8%: shrinking the input made the model's output expand to compensate. The authors' conclusion was blunt — "compress more" is not a reliable production heuristic. If you only ever measure input tokens, you can optimise yourself into a larger bill.

Why code resists compression

The mechanism explains the numbers. Most compressors work by extractive or perplexity pruning: they drop the tokens a small model finds most predictable, on the reasonable theory that predictable tokens carry the least information.

In prose, the predictable tokens are filler, and dropping them is nearly free. In code, the predictable tokens are frequently the load-bearing ones — exact file paths, JSON keys, error codes, function signatures. The most cited example in this space is a pruner stripping auth.ts and jwt out of a stack trace precisely because both were predictable from the surrounding path and error text. Those were the two tokens the agent needed to fix the bug.

A dropped adjective rarely changes a question-answering result. A single dropped character in a file path turns a correct patch into one that will not apply. This is also why long-context and QA benchmark scores do not transfer as evidence for coding work: compression degrades prose gracefully and code catastrophically, and the benchmarks that get quoted are measuring the graceful case.

The evaluation gap underneath all of it

The deeper problem with LLM token compression claims is not exaggeration — it is that almost nobody reports downstream task success at a stated compression ratio.

The canonical compressors report fidelity scores and long-context QA benchmarks. A small 2026 wave of coding-specific papers does attempt task success, and none of it is conclusive yet: one shows a 1.2–1.4 point gain on 500 instances with no confidence intervals (comfortably inside run-to-run noise), another reports a 33% token cut with no pass@1 numbers at all, and the one paper that runs a proper paired significance test correctly reports its own result as non-significant.

So the honest state of the field: compression demonstrably reduces tokens, and whether it preserves the ability to finish a real task at a given ratio is largely unmeasured. If a tool cannot tell you the second number, a quoted percentage on the first is not a savings estimate — it is a compression ratio wearing a dollar sign.

Caveat, stated plainly: the vendor figures above are self-reported and not independently verified, and the academic results are preprints. We are repeating them as published, with their own framing intact.

What to do with this

Three practical conclusions:

  1. Ask what was compressed before you believe a percentage. JSON, logs and tool schemas are real wins — just not the same win as compressing your code. Tool-schema bloat in particular is worth attacking on its own terms; we wrote about that in tool-schema bloat.
  2. Measure output tokens too. The 1.8% result is the warning: input-only optimisation can quietly cost you money.
  3. Compression is not the only lever, and often not the biggest. Shrinking the input reduces tokens per call. Running the right-sized model changes the price of every one of those tokens, and the gap between a frontier model and a small one on a trivial task is frequently 20–100×. Those compound, and most teams have the second one sitting untouched in their code — see how to reduce LLM token costs for the full set.

The reason the second lever goes untouched is rarely discovery. It is that nobody wants to ship a cheaper model without evidence it still works — which is a verification problem, not a compression problem. We wrote up how to settle it in how to prove a model swap holds quality.

If you want to know which of these your own code is paying for, run a free scan — it finds the call sites and gives you an honest dollar estimate per site, with every assumption labelled.

EB
erabot.ai

Scan any codebase. See exactly what you spend on AI. Get one-click fixes.

No spam. Only launch updates and cost optimization tips.

Product

  • Product
  • CLI
  • Pricing
  • Changelog
  • Docs

Resources

  • Example audit
  • Blog
  • Case studies
  • Eval

Company

  • Security
  • Privacy
  • Terms
  • Support

© 2026 erabot.ai. All rights reserved.