You Cannot Cut AI Model Cost Until You Can Measure What Breaks

The easy part is agreeing, the hard part is proving it

Not every task in our product needs the most expensive model available. Almost everyone I say that to agrees immediately. Then nothing happens, because the next question stops the conversation. If I swap in a cheaper model, does the product get worse?

Without an answer to that, the cheaper model is a guess. A guess about accuracy is not something you ship into a regulated workflow, and it is not something a customer will accept if you are wrong. So the cost work does not start with models at all. It starts with building something that can tell you whether accuracy moved, in either direction, before anything reaches a real conversation.

That thing is a labeled test set. Getting one is harder than it sounds, because for most of these decisions there is no ground truth sitting anywhere. Nobody wrote down the right answer. The bot made a pick, the conversation moved on, and that is all the history you have.

Building a labeled set when nobody wrote down the answer

The approach we settled on is a voting panel. For each example, four strong models answer independently. If all four agree, we take that as the label. If they split two against two, we drop the example, because a genuine split means the case is ambiguous and no label we assign would be honest.

The interesting case is three against one. There, we hand the whole thing to a separate state-of-the-art model, and we give it everything, all four answers and the reasoning behind each of them. If that model lands with the majority, we keep the label. If it does not, we drop the example.

The point of the drop rule is not strictness for its own sake. A test set full of ambiguous cases produces accuracy numbers that move for reasons unrelated to model quality, and then you are measuring noise and calling it a regression. What we want in the set is examples we are almost certain we have labeled correctly.

Dropping an example does not mean the product never sees cases like it. In production the models still have to answer, and they do. What catches those is QA reviewing decisions after the fact, not the test set. The test set is a measuring instrument, and its job is to be trustworthy rather than complete.

Choosing which examples are worth labeling

Running four strong models plus a tie-breaker over every candidate row costs real money, so labeling everything was never an option. That constraint forced a better design than an unlimited budget would have. The set gets built in three parts, and each part exists for a different reason.

The first part is a representative core. Rows are allocated across cells of decision type and outcome in proportion to real production volume, so a policy that fires constantly gets proportionally more rows than one that fires twice a month. That preserves the real base rate instead of an artificial balanced split. Inside each cell, random picking would still hurt us, because random sampling happily grabs fifty near-identical rephrasings of the same complaint and calls it coverage. So we embed the conversations and run a greedy diversity selection over that embedding space, spreading the picks across the actual variety of conversations in the cell. This part of the set is deliberately blind to difficulty. Its only job is to look like production.

The second part is a hard slice, chosen by a purpose-built difficulty model. This is a small local classifier, not one of the models under test, and it costs nothing to run. We trained it on labels we already had rather than buying new ones, using the panel's own agreement pattern as the signal for what counts as hard. It scores every candidate row on features that correlate with being genuinely difficult to judge. Whether the conversation is a near duplicate of one that got a contradictory verdict somewhere else. Whether a sibling policy under the same parent also fired. How many policies get evaluated on the ticket at all. Whether the conversation is in another language. How long it runs. How historically hard that specific policy has been. And how hard the nearest neighbours in embedding space turned out to be. Those scores become hardness bands, and the hard rows get pulled in with a floor guaranteeing every policy contributes at least one hard positive and one hard negative, plus a cap so no single policy eats the whole budget.

The third part is a set of guarantees on top. Every policy gets a real positive and a real negative wherever production logged both, so even a rare-outcome policy is graded on both classes. Human-reviewed tickets are folded in directly as ground truth, no panel needed. The ones where a human found production wrong get pinned to the hardest band regardless of what the difficulty model thought, because a confirmed production failure is the most valuable row in the set and I do not want a model score demoting it.

One correction that matters. Because the hard slice deliberately oversamples difficult rows, the raw pass rate on this set would understate real quality. So each cell carries a weight based on how much of the population it actually represents, and accuracy is computed against those weights. The difficulty boost buys sensitivity without shifting the estimate.

The same method does not fit every component

We built this for the first-pass filter stage, and it worked. When we applied it to the option-picking stage, the difficulty classifier had essentially no discriminating power, so we turned that part off rather than shipping a component that added cost and no signal.

What replaced it was self-consistency. We re-run production's own model on a question it has already answered and measure how often it agrees with itself across runs. Where a question is crisp, meaning high self-agreement, low overlap between the wording of neighbouring menu options, and almost no answers falling outside the menu, we score by exact match and skip the panel entirely. Only the soft questions get the full voting panel and the tie-breaker. That saved a large share of the labeling budget on a component where the expensive path was buying nothing.

The transferable lesson is that the components share the allocator, the diversity selection, the weighting formula and the drop rule, and differ on everything downstream of that. Assuming one evaluation recipe generalizes across your whole pipeline is a good way to spend money on a measurement that does not measure anything. Notch runs a lot of these decisions across very different accounts, and you can see how the stages fit together on the product side.

What the test set actually buys you

Once the labeled set exists, the cost work becomes cheap and fast. I can run any candidate model against it in an afternoon and see exactly what happens to accuracy.

I have my own priors going in. Gemini models are strong at classification for the price, and Luna is unusually cheap for the quality it delivers. But priors only decide what I test first. The test set decides what wins.

The real gain is that I stop evaluating single models and start evaluating combinations. Once I have per-example results for every candidate, an algorithm can check the combinations and cascade designs mechanically. Run two cheap models on everything, and only pay for a stronger model on the small share of cases where the cheap ones disagree. Two cheap calls together still cost less than one expensive call, and the expensive third call only fires on the minority of decisions that genuinely need it. Two models that fail in different ways landing on the same answer is a strong signal, and the test set is what proves how strong.

I evaluate those combinations on three axes at once using a Pareto front, so accuracy, cost, and latency all count. A design that beats another on all three wins outright. Anything surviving on the front is buying something real with whatever extra it costs. Our thinking on mixing model tiers this way is written up in more detail in our hybrid model playbook.

The pricing bug that a price cut uncovered

We got one thing badly wrong, and it is worth telling because of how it surfaced.

A stale price alias in our tooling had one model inheriting an older model's price, several times cheaper than the real thing. That made a straight model swap look about three times cheaper than production. We had already said that number out loud before anyone noticed. At the correct price the same swap saved almost nothing.

What caught it was not a review. Luna's price came down, which meant re-pulling the price table, and the stale alias fell out immediately. We were lucky in the timing, and I did not want to depend on luck again, so we built the price and model checks to run automatically. When a new model appears or a price changes, the system picks it up and replaces what it needs to.

The fix for the claim itself was not walking the pitch back and stopping there. We redesigned around a cascade at the corrected price, which restored the savings for real. The savings came from the cascade design and the routing rules, not from swapping one model for another.

Cost, accuracy and latency is a product call

People expect the research to output one recommended answer. It does not, and I think that is correct.

What matters most depends on the product. In voice, latency dominates, because a delay a user can hear is a worse failure than a slightly higher bill. In most other places cost matters more, as long as latency stays inside the budget. Those are different optimal designs from the same data.

So what the Pareto front produces is the set of designs where the trade-offs are genuine, and product decides which one to take. The research does not make the choice. The research makes the choice available and makes the consequences of each option visible before anyone commits.

When two cheap models agree and both are wrong

The obvious objection to any cascade is that trusting agreement fails when both cheap models are confidently wrong together. It happens. The test set tells us how often, per decision type, and the rate is low. Lower than the error rate we run in production today.

That last comparison is the bar I hold every component to. It does not need to be perfect. It needs to be better than what we are running now.

The other reason a small error rate is survivable is that these components do not stand alone. Our pipeline has stages that re-judge earlier decisions, so a component can be wrong occasionally and the end result can still be right. That only works if you know the error rate. Guessing at it and hoping the next stage cleans up is how you ship a quiet regression. The same logic drives how we test conversations end to end rather than only at the component level.

Where a cascade does lose to production on a specific decision type, we do not hide it or average it away. We route that one decision back to the stronger model and keep the cheap path everywhere else.

What I would tell a team starting this

Someone asked me recently how to cut model cost when they have no labeled test set. My honest answer was that they cannot. Not safely. Without labels they have no way to know whether the cheaper model made accuracy worse, and finding out from customers is the most expensive way to learn it.

Build the measuring instrument first. The cost savings are sitting there and they are large, but they only become available once you can prove what you did not break.

‍