erabot.ai
ProductPricingCase studiesEvalBlogDocs
Book a demo

The caching trap: when a cost fix silently breaks your token counters

Prompt caching is, more often than not, the single biggest cost win we find in a codebase. If your app re-sends the same large prefix — a system prompt, a block of tool schemas, a growing transcript — on every call in a loop, you are paying full price for tokens the model has already seen. Turn on caching and those cached reads bill at roughly a tenth of the input price. On a busy agent loop that is frequently the difference between a comfortable bill and an alarming one.

So we recommend it constantly. And there is a side effect almost nobody warns you about — one that can quietly break production in a way your tests will never catch.

What caching actually changes

Start with how usage is reported. Anthropic's Messages API returns token usage as three separate input counters, and they are disjoint — each token is counted in exactly one of them:

  • input_tokens — uncached input tokens, billed at the normal input rate
  • cache_creation_input_tokens — tokens written into the cache on this call (billed ~1.25x input)
  • cache_read_input_tokens — tokens served from the cache (billed ~0.1x input)

Before you enable caching, only one bucket matters: input_tokens holds the entire prompt. It is a faithful proxy for "how big is this request." A lot of code quietly relies on that.

The moment you turn caching on, that stops being true. The cached prefix moves out of input_tokens and into cache_read_input_tokens (or cache_creation_input_tokens on the first call). On a cache hit, input_tokens can report something close to zero while the real prompt is still tens of thousands of tokens filling the model's context window. The bill goes down — great, that was the point — but the number called input_tokens now means something fundamentally different. It no longer measures the size of your prompt. It measures only the uncached remainder.

Why that breaks things

Here is the trap. Caching is a cost optimization, but input_tokens is not only read by cost code. In most codebases it is quietly load-bearing for decisions that have nothing to do with billing:

  • Context-window guards that elide or summarize history "when the prompt gets too big"
  • Spend caps and budget stops that tally input tokens
  • Rate limiters keyed on token throughput
  • Truncation logic that trims inputs to stay under a limit
  • Any "am I close to the context limit?" check

Every one of those reads input_tokens as a stand-in for prompt size. After caching, that stand-in is wrong — not by a little, by the size of the entire cached prefix. And nothing errors. There is no exception, no warning. The number is still an integer, still plausible, still there. It just quietly means the opposite of what the code assumes: a huge cached prompt now looks tiny.

So the guard that was supposed to fire never fires. The budget that was supposed to stop never stops. The truncation that was supposed to trim decides there is nothing to trim.

The incident

We hit exactly this on one of our own agent-evaluation runs. We had a context-elision guard whose whole job was to keep the prompt under control: when input tokens crossed a threshold, it would summarize and drop older turns. Standard stuff.

We turned on prompt caching to cut the cost of the run — our own number-one recommended fix. And the guard silently stopped working. Because the large, stable prefix was now being served from cache, input_tokens came back small on almost every call. The guard looked at that small number, concluded the prompt was nowhere near the threshold, and never elided anything. The real context kept growing in the background, unbounded, right up until it hit the model's hard limit.

The part that makes this genuinely dangerous: it was invisible to our tests. Unit tests mock the API, and the mocks returned a fixed token count — the pre-caching behaviour. In the mocked world the guard worked perfectly. It was only on a real-API run, where the three counters actually split the way the live API splits them, that the bug appeared. The thing that broke was only observable against the real thing.

The fix

The fix is small once you see it, and it is two parts.

First, the number. Anything that needs the true size of a prompt must read the sum of all three input counters, not input_tokens alone:

true_prompt_tokens = input_tokens
                   + cache_creation_input_tokens
                   + cache_read_input_tokens

That sum is invariant to whether caching is on or off — it is the real size of what you sent. Every budget check, context guard, rate limiter, and truncation decision should key off that sum, never off input_tokens by itself.

Second, the audit. When you turn on caching, find every place the code reads input_tokens (and the equivalent field in whatever SDK you use) and ask of each one: is this a billing number, or a size/limit decision? Billing can stay as it is — you do want to see the cache discount. Every size decision has to move to the sum. This is a ten-minute audit and it is the difference between a clean cost win and a latent production incident.

The real lesson

The specific bug is about one API's counters, but the shape of it is general, and it is the reason we are careful about how we present cost fixes.

A cost optimization is a behaviour change. Caching does not just make the same thing cheaper; it changes the meaning of a number that the rest of your system was reading. Whenever a fix changes what a measurement means, every consumer of that measurement is a potential casualty — and the ones that hurt are the consumers you forgot were reading it.

Two habits fall out of that.

Measure the thing you are actually deciding on. The decision was "is the prompt too big?" The right measurement is prompt size — the sum of the counters — not input_tokens, which merely correlated with prompt size until caching broke the correlation.

Trust real runs over mocks for anything involving usage numbers. Mocks encode your assumptions, and this is precisely a bug in the assumptions. A mock that returns a fixed input_tokens will happily pass while production quietly drifts. If a change touches token accounting, it needs at least one real-API run before you believe it.

This is also why, when erabot recommends caching as a fix — and it does, constantly, because it is usually the biggest single lever — it flags this caveat alongside it. A fix that saves 60% on your bill and silently disables your context guard is not a good trade if you find out the hard way. Saving money and knowing what your numbers mean are the same discipline.


If you want to know where your own code re-sends the same context uncached — and get the caching fix with this caveat attached — erabot scans your repo and shows you, with an honest dollar estimate per call site. Free at erabot.ai.

EB
erabot.ai

Scan any codebase. See exactly what you spend on AI. Get one-click fixes.

No spam. Only launch updates and cost optimization tips.

Product

  • Product
  • CLI
  • Pricing
  • Changelog
  • Docs

Resources

  • Example audit
  • Blog
  • Case studies
  • Eval

Company

  • Security
  • Privacy
  • Terms
  • Support

© 2026 erabot.ai. All rights reserved.