Can a cheaper model do the job? How to actually prove it
"Just use a cheaper model" is the most common piece of LLM cost advice on the internet, and the most dangerous — not because it is wrong, but because almost nobody verifies it properly. A cheaper model that is 90% as good is a great deal on a task where 90% is fine, and a disaster on the one where the missing 10% is the whole point. The difference between those two cases is not something you can eyeball. It has to be measured. And most of the ways people "measure" it do not actually prove anything.
Here is how to tell whether a swap holds — and why the quick version usually does not.
The two ways people fake the proof
Version one: swap the model, read a handful of outputs, decide they look fine, ship. This is vibes. It feels like evidence because you looked at real outputs, but a few cherry-read examples tell you nothing about the tail — and the tail is where regressions live.
Version two, the more sophisticated fake: run the old and new model on ten or twenty examples, have a strong model judge which output is better on each, and if the new one is not worse, call it proven. This looks rigorous. It has a dataset, a judge, a verdict. It is still not a proof, for two independent reasons.
Reason one: the judge is noisy
LLM-as-judge is not a stable instrument. A 2026 study of two frontier judges across 29 tasks found their preference verdicts flip about 13.6% of the time from run-to-run noise alone — same inputs, same judge, different answer — with some questions flipping more than 20% of the time and one as high as 56%. Agreement between two different judges was only 76% (Cohen's κ around 0.51). Reword the judge prompt into something semantically equivalent and the majority verdict changes in a quarter of cases. Recovering a verdict stable enough to trust took roughly eleven repeated runs per comparison.
On top of that noise sit well-documented biases: judges tend to prefer the first option they are shown, prefer longer answers, and prefer outputs from models like themselves. So a single judge call per example — the obvious way to build a quick proof — is a coin weighted by order and verbosity. Your "no regression" verdict might reverse on a re-run you never did.
Reason two: ten examples cannot carry a conclusion
Even with a perfect judge, a handful of examples cannot support a confident claim. The statistics are unforgiving: to say "the new model is within 5 percentage points of the old one" at 95% confidence, on a metric around 80%, you need roughly 250 examples. Want a tighter margin? The number climbs fast.
What does ten examples actually buy you? In one small evaluation, nine models over 33 questions produced confidence intervals about 30 points wide — wide enough that most pairwise comparisons were statistically uncallable. The honest reading of a green check on fifteen examples is not "it holds." It is "we have no idea." A proof built on a sample that small is not a weak proof; it is a decoration.
How to actually do it
A real non-inferiority check — "the cheaper option is not meaningfully worse" — is buildable. It just requires respecting the two problems above instead of wishing them away.
1. Prefer objective verifiers wherever the task allows one. If the output is a label, check it against the known answer. If it is JSON, validate the schema. If it is code, run the tests. An executable check gives you a correct-or-incorrect answer without a model probabilistically guessing at quality, which sidesteps the judge-noise problem entirely. A surprising fraction of "subjective-looking" tasks — extraction, classification, routing, structured generation — have an objective core you can grade this way. Use it for as much of the task as you can, and reserve judging for the genuinely subjective remainder.
2. When you must use a judge, treat it like the noisy instrument it is. Fix the judge and its prompt up front, control for answer order, and take a majority vote over repeated runs rather than trusting a single call.
3. Compare in pairs. Run the old and new model on the same inputs and compare per example, not in aggregate. A paired design cancels much of the variance that comes from examples simply being easy or hard, and reaches the same confidence with fewer samples than comparing two separate averages.
4. Pick a margin, and be willing to say "not yet." Decide in advance how much quality you are willing to risk — the non-inferiority margin. Then report the actual confidence interval against it. If the interval is too wide to clear the margin, the correct output is not a verdict; it is "we cannot certify this on the data we have — send more traffic." A tool that refuses to certify when it should is worth more than one that always prints a green check.
The reframe that makes this tractable
There is a reason the industry's dominant approach to "use a cheaper model" — runtime routing, where a system guesses per request whether a cheap model will do — keeps disappointing. The RouterArena benchmark (ICLR 2026) tested twelve routers on 8,400 queries and found that all of them fall short of the best achievable cost-quality tradeoff, most cluster near the expensive model's cost because they over-select it out of caution, and their robustness is "uniformly low." Predicting, before you see the answer, whether the cheap model will be good enough is genuinely hard, and the field has not solved it.
But proving a swap holds does not require solving that. The honest version of the problem is retrospective, not predictive: take examples whose correct answers you already know, shadow-test the cheaper model and the rewritten prompt against them, certify that it held, and only then switch. You are not gambling on each future request. You are making one well-measured decision on known data and carrying it forward. That is a far easier and far more honest problem than live routing — and it is the right shape for a cost decision you are going to have to defend later.
Why we are so particular about this
This is the discipline the entire idea of a trustworthy cost tool rests on. It is easy to build something that always tells you "yes, the cheaper model is fine" — that tool would be popular and useless, because the one time it is wrong is the time that matters. A cheaper model is a candidate until it is shadow-tested, and the value is not in suggesting the swap — anyone can read a leaderboard — but in proving it holds on your task, with real verifiers, enough samples, and the honesty to say when the data is too thin to tell. Finding the swap is a five-second leaderboard glance. Proving it is the hard part, and the whole job.
erabot's verified audit shadow-tests each cost fix against your real traffic before recommending you ship it — and marks every swap as a candidate until it is proven. See how it works at erabot.ai.