You can move an agent to a cheaper model without regressions when you prove it rather than assume it: replay one frozen test suite against every candidate model, score each with the same judges against the same bar, require every repeated run to pass, and read the result per agent with the sample size shown. The cheapest model that clears that test, and is not measurably worse than the one you run today, is the move.
Why teams stay on the biggest model
Most teams pick the largest model by default when they build an agent, then never revisit the choice. The reason is rarely that the large model is needed. It is that nobody can prove a smaller one is good enough, and the cost of a quiet quality drop in front of customers feels worse than the cost of the bill.
That leaves money on the table in both directions. Many agents do not need the most capable model at every step, but some do, and a blanket switch ships the wrong model to exactly the work that matters most. The answer is not a hunch about which model "sounds fine". It is an experiment designed so the only thing that changes is the model.
What a fair comparison holds fixed
A model comparison is only as good as its controls. If the prompts, the graders or the bar differ between two runs, you are comparing experiments, not models.
| Hold fixed | Why it matters |
|---|---|
| The test inputs | A frozen version of the suite, so every model sees the same prompts in the same state |
| The judges | The same graders at the same versions, so a score means the same thing on every side |
| The bar | A pass rate chosen before the results, not after, so the verdict cannot be tuned to a winner |
| The repeats | The same number of independent runs per model, so no model gets extra chances |
| The baseline | The model you run today, declared up front, so savings and regressions are measured against something real |
| The price source | Cost computed from each run's own token counts and a recorded price per model, never an assumed rate |
If any of these drifts during the test, the honest outcome is no verdict at all. A comparison of runs that measured different things is the most convincing wrong answer you can produce.
Six ways a cheaper model looks safer than it is
Each of these produces a number that reads like a fact and is not.
- One lucky green run. Models are non-deterministic and tools are flaky. A model that passes once and fails twice has not passed. Count a model as passing only when all of its repeated runs pass, and count a test item as passed only when every repeat of it passed.
- The workspace average. A cheaper model can lose a point of pass rate on one agent and a large share on another. The average of the two hides the second, which is the one that hurts. Decide per agent.
- Mixed questions. A pass rate averaged across a helpfulness judge and a safety judge is not one bar. Narrow to the metric you care about before you quote a number.
- A small sample. Two pass rates a few points apart over a few dozen items are often not different at all. Report a confidence interval with every rate, and treat overlapping intervals as "not distinguishable at this sample size".
- A missing price read as free. If you do not know what a model costs, it is not the cheapest option. Leave its cost unknown until a real rate is recorded.
- Traffic you cannot attribute. Production traces that used two models, or recorded none, do not belong to either model's bar. Keep them in their own bucket rather than folding them into a winner.
A method that holds up
The steps below turn the controls and traps above into a repeatable decision.
- Build the suite from your own failures. A public benchmark tells you how a model does on somebody else's problems. A golden set built from the failure modes your agent actually hit tells you whether a cheaper model survives yours.
- Freeze everything except the model. Pin the suite version, the judges, the number of repeats, the bar and the baseline once, and reuse them for every candidate.
- Estimate the cost before you spend. Every candidate costs repeats times items in model calls. Preview the total first, note which parts are measured and which are assumptions, and set a budget ceiling.
- Run each candidate several times. Three independent runs per model is a sensible default. A model that cannot pass all of them is out, whatever its average.
- Read the verdict per agent, with the uncertainty. For each agent, name the cheapest model that clears the bar, the pass rate and sample size behind it, and whether the interval's lower bound also clears. If it does not, the claim is plausible but not yet proven, and the fix is more test items.
- Look at where it breaks. For any candidate that regresses, read the items the baseline passed and the candidate failed: the input, the expected answer, what the candidate actually said, and the judge's reason. Group them by failure mode.
- Ship behind a gate, then watch production. Make the switch through the same release gate as any other change, so a regression is blocked rather than discovered. Afterwards, slice the production pass rate by model to confirm the result held on real traffic. This is the same discipline a self-healing agent harness applies to every fix.
The output of this process is a sentence you can defend in a review: for this agent, this model clears our bar at this pass rate over this many items, and costs less than what we run today. For that other agent, it does not.
Reading the result as a frontier
With several candidates, a chart makes the decision faster than a table. Put cost per case on one axis and quality on the other. The line through the best points is the efficient frontier: the most quality available at each price. Every candidate then falls into one of a few roles.
| Role | What it means |
|---|---|
| Your model now | The incumbent every move is measured against |
| On the frontier | Nothing cheaper is at least as good, so there is no free saving here |
| Recommended | The cheapest model that is cheaper than yours, clears the bar, and is not measurably worse |
| Overpaying | Something cheaper is at least as good, so the extra spend buys nothing |
| Below the bar | Cheap, but not good enough to consider |
| Quality unknown | Its cost is known but there is no quality signal yet, so it is shown and not placed |
The confidence interval acts as a noise band. If the cheaper model's interval overlaps yours, the switch is a no measurable quality loss move. If the intervals separate, the difference is real, in one direction or the other, and a model that is distinguishably worse is not a safe move however much it saves.
Questions
How do I know a cheaper model is good enough for my agent?
Replay the same frozen test suite against both models, score them with the same judges, and compare pass rates against a bar you set in advance. Require every repeated run to pass, and read the result per agent with the sample size and confidence interval beside it.
Why run each model more than once?
Because a single green run can be luck: a non-deterministic model, a flaky tool call or a generous grading pass. Requiring k independent runs to all pass means a model that passes once and fails twice reports a fail, not a pass with variance.
Can a cheaper model be right for one agent and wrong for another?
Yes, and it often is. A workspace average can hide an agent that loses a large share of its pass rate on the cheaper model. Read the verdict per agent and switch only the agents that clear the bar.
What if the two pass rates are very close?
Then check the confidence intervals. If they overlap, the sample cannot tell the models apart, which supports a no measurable quality loss switch. If the cheaper model is distinguishably worse, it is not a safe move however much it saves. Adding more test items is the way to a firmer answer.
What if a candidate model has no known price?
Treat its cost as unknown, never as zero. An unpriced model can still be judged on quality, but it cannot be ranked as the cheapest option until you record a rate for it.
Sources
- Neens Docs, Model sweeps: frozen golden set, pinned conditions, pass^k per arm, cost preview.
- Neens Docs, Sweep decisions: the quality bar, per-agent verdicts, confidence intervals, unpriced arms, regression drill-down.
- Neens Docs, Cost and quality frontier: frontier roles and the noise band.
- Neens Docs, Model comparison: pass rate by model and by agent × model on production traffic.