LLM Cost Optimization: the complete guide
TL;DR: Almost every large LLM bill comes from the same short list of code patterns — the wrong model for a simple task, the same context re-sent uncached in a loop, oversized prompts, uncapped agent loops, un-batched calls, and missing output limits. None of them appear on a usage dashboard labelled "the problem"; they live in the code. This guide names each pattern, shows how to estimate what it costs you honestly, and — the part most advice skips — how to prove a fix actually holds quality before you ship it.
Why your bill is a code problem, not a dashboard problem
When an AI bill spikes, the first instinct is to open the provider dashboard. It will tell you what you spent, by model and by day. It will not tell you why, and it cannot tell you how to spend less — because the decisions that set your bill were all made in code: which model each call uses, whether the stable part of the prompt is cached, how much context you send, how many times a loop runs. A dashboard is a thermometer, not a diagnosis. (We wrote about this gap in detail in Observability tells you what you spent; it can't tell you how to spend less.)
Runtime tools — AI gateways and routers — sit one layer closer, rerouting requests as they happen. They help, but they move the lever out of your code and into infrastructure, and they decide per request without ever telling you which lines were wasteful. The durable fix is upstream, in the code, where the pattern lives and compounds on every run. So that is where this guide looks.
The six patterns that drive almost every overspend
1. Uncached repeated context — usually the number one leak
If a call sits inside a loop or an agent turn-loop that re-sends the same large prefix every iteration — a system prompt, a block of tool schemas, a growing transcript — you are paying full input price for tokens the model has already seen. Provider prompt caching bills those repeated reads at roughly a tenth of the input rate. On a busy agent this is frequently the single biggest line item: Stanford research (cited widely in 2026) attributes a majority of agent inference cost to re-sent context alone, which is the subject of The 62% problem.
The fix is to cache the stable prefix. But caching is a behaviour change, not a free lunch: turning it on changes what the input_tokens counter means, which can silently break any budget, context, or truncation logic that reads it. Before you ship a caching fix, read The caching trap — it is a real incident, and the two-line fix that prevents it.
2. A frontier model on a simple task
After uncached context, the most common leak we find is an expensive model doing a narrow job: classification, routing, a yes/no, extraction, a short label. Those tasks almost never need a frontier model, and the price gap between tiers is often 20x to 100x. We found exactly this in OpenAI's own Swarm framework — a name-extractor running on a model roughly 65x more expensive than it needed to be.
The catch: "cheaper" is not automatically "fine," and sometimes a cheaper model is actually more expensive because it rambles — see The verbosity trap, where a cheaper model cost 29% more. So a model swap is always a candidate until it is verified (more on that below).
3. Uncapped loops and no budget
An agent or retry loop with no turn cap, or a batch job with no aggregate spend ceiling, is an open tap. One malformed input or one stuck agent can run the bill up with nothing to stop it. The fix is cheap and defensive: a maximum-turns cap on every loop, and an aggregate cost stop on every batch.
4. Oversized / stuffed context
Sending a whole file, or a full conversation history, on every call when a slice or a retrieval would do is pure waste that scales with your traffic. This is especially common in RAG pipelines, where a default "no reranker" setting can stuff dozens of unranked chunks into every answer call. We took apart two structural versions of this in a production system in the RAG platform case study, and the general mechanics are in The hidden cost of long context windows. The fix is to trim, retrieve, or summarise once rather than re-sending everything.
5. No batching where calls are batchable
Many independent calls fired one at a time in a loop — scoring items, classifying rows, generating variants — can often be a single batched request or one shared-context call, at a fraction of the overhead. If the calls do not depend on each other's output, they are candidates for batching.
6. Missing output limits / runaway thinking
No max_tokens means the model can run long on every call; an extended-thinking or reasoning-effort budget set higher than the task needs burns tokens invisibly. Set an explicit output cap, and tune the thinking budget to the job.
Where the decision actually hides: config, not the call site
Here is the trap that makes most cost scans useless: the model is rarely a string literal at the call site. It is a config value, an environment default, or buried behind an SDK wrapper — so a shallow scan reports "model: unresolved" and lowballs the whole codebase by a factor of a hundred. The real cost of an agent lives in which tier each call uses, and you cannot see it without resolving the config.
Two of our teardowns show both sides of this. gpt-researcher does it right — an explicit fast/smart/strategic tiering where each job gets the model it needs. MetaGPT shows the failure mode: the real model decision lives in an example config everyone copies, and that config had gone a generation stale. When you audit your own stack, resolve the model through the config and the wrappers, or you are not measuring the thing that actually sets your bill.
Estimating the damage — honestly
The honest way to size a leak is to look for real data first. Before estimating from static code, check the repo for actual usage records — token logs, exported provider dashboards, trajectory dumps. Real numbers beat a static guess every time and often surface issues static analysis cannot.
When you do estimate from code, infer volume where you can (calls per request times requests per month; loop turns times runs) and then — this is the rule that keeps the number trustworthy — mark every assumed volume as assumed. A finding with a clearly-labelled assumption is useful; an invented certainty is not. We ran this discipline across 15 open-source AI products: the percentages are exact and checkable; the monthly dollars depend on your traffic, and we said so every time.
The hard part: proving a fix holds
Finding a cheaper option is the easy half. Proving it does not quietly degrade quality is the half almost everyone skips — and it is where the real engineering is. You cannot tell from static code that a cheaper model holds quality on your traffic, so every swap stays a candidate until it is shadow-tested.
And the quick "proof" most people reach for does not hold up: a single LLM-judge call on a handful of examples. LLM judges are measurably noisy, and a sample of ten or twenty examples is statistically indefensible. The full argument, with the numbers, is in Can a cheaper model do the job? How to actually prove it, and the machinery we use to do it properly — objective verifiers, paired designs, non-inferiority margins — is in Inside the shadow harness. The discipline matters because evals lie if you let them: we once published a retraction when our own eval reported a perfect score it had not earned.
A practical order of operations
If you are optimising a real codebase, do it in this order — biggest, safest wins first:
- Inventory the call sites and resolve the models (through config, env, and wrappers). You cannot fix what you have mis-measured.
- Find the uncached loop. It is usually the largest single number, and caching the stable prefix is the highest-leverage fix — applied with the counter caveat.
- Right-size the obvious model mismatches (classification, routing, extraction on a frontier model) — as candidates, to be verified.
- Cap loops and outputs. Cheap, defensive, done once.
- Trim context and batch where a slice or a single request would do.
- Verify every behaviour-changing swap before it ships. A proven 30% cut beats an unproven 60% one.
Scan your own code
You can run this inventory by hand, or let erabot do it: it scans your repo (or any public one), resolves the models through your config and wrappers, estimates the monthly cost per call site with assumptions marked, and ranks the findings by dollar impact — free. The verified audit then shadow-tests each fix to prove it holds quality before you ship it. Start at erabot.ai.