Business & Money
How Do You Keep AI Costs Under Control as a Product Grows?
By Jim Vernon, Editor, AI Intelligence International · Published 27 August 2026 · Reviewed against our editorial standards · About the author
AI-backed features have a cost structure most software teams are not used to: a real marginal cost per use that rises with engagement. Growth that would once have been free now shows up directly on the bill.
This article covers the controls that matter — measurement, routing, caching, limits and pricing — in the order they usually pay off.
Key takeaways
- Measure cost per user action before optimising anything; most teams cannot name their most expensive endpoint.
- Routing cheap cases to smaller models is usually the largest single saving available.
- Caching works better than expected because real user inputs repeat more than teams assume.
- If a heavy user can cost more than they pay, the pricing model is the bug, not the usage.
What should you measure first?
Cost per user-visible action, not cost per API call. Users do not trigger calls, they trigger features, and a single feature may fan out into several calls of very different sizes.
Attribute every call to a feature and a customer at the point of the call. Retrofitting attribution later is painful, and without it your bill is a single number you cannot act on.
Then rank features by total spend, not by unit cost. The expensive feature is often not the one with the highest per-call cost but the cheap one everybody uses.
Where does routing save the most?
In almost every product, a minority of requests genuinely need the strongest model. Classification, extraction, short rewrites and routine formatting are usually handled well by substantially cheaper models.
Build the routing decision on a checkable property — input length, task type, customer tier, confidence of a first-pass classifier — rather than on a guess, and log which route each request took.
Measure quality per route separately. The failure pattern to avoid is a global quality drop that nobody notices because the aggregate metric is dominated by the easy cases.
Does caching actually help with generated output?
More than teams expect. Real inputs cluster heavily: the same document summarised repeatedly, the same handful of questions, the same template with a changed name.
Cache on a normalised input key and give entries a sensible lifetime. Even a 20% hit rate on your highest-volume endpoint is a direct proportional saving with no quality cost.
Watch for personalisation leaking into cache keys, which both kills the hit rate and risks serving one customer's content to another. Keys must include every input that can change the output.
How do you set limits without wrecking the experience?
Set per-account and per-key ceilings from day one, with alerts well below them. The scenario that produces a shocking bill is almost always a loop, a retry storm or an integration bug, not organic growth.
Prefer degradation to hard failure: fall back to a cheaper model, a shorter output or a queued response rather than an error, and tell the user what happened.
Rate-limit by cost, not by request count. Ten small calls and one enormous one are very different events that a request counter treats identically.
When is the pricing model the real problem?
Whenever a plausible heavy user costs more to serve than they pay. Flat-rate pricing on a feature with real marginal cost is a bet that usage stays average, and popular products break that bet.
The fixes are familiar: usage-based components, credits included in a tier, fair-use ceilings, or a higher tier that heavy usage naturally pushes users into.
Model this at your worst realistic user, not your median. If the top 1% of accounts by usage are unprofitable and growing, no engineering optimisation will outrun it.
What about prompt and output size?
Prompt bloat is the quiet cost driver. Long system prompts, whole documents pasted where an excerpt would do, and full conversation history resent on every turn all multiply against every single call.
Trim context to what the task needs, summarise or window long histories, and cap output length explicitly. These changes are cheap to make and often cut spend by a third with no quality change.
Re-measure after each change. Some trims cost quality in ways that only show up in specific input types, which is why per-route quality tracking matters.
Worked example: cutting a bill by 61%
A document-workflow product was spending roughly 18,400 per month on inference with about 3,000 active accounts, and the figure was rising faster than revenue.
Attribution showed 71% of spend came from one endpoint: an automatic summary regenerated every time a document was opened, not only when it changed. Adding a content-hash cache removed most of those calls at a hit rate of 63%.
Second, a classification step used the same frontier model as the summary. Moving classification to a small model cost roughly a point of accuracy on a blind sample of 400 cases and reduced that line by about 90%.
Third, the system prompt had accumulated to around 1,900 tokens, much of it obsolete instructions. Trimming it to 620 cut every remaining call.
Combined, monthly spend fell to about 7,100 with no measurable change in user-reported quality. They then added a fair-use ceiling at ten times median usage, which affected four accounts, all of whom were on the lowest tier.
Frequently asked questions
Is a cheaper model always worth trying first?
It is the highest-yield experiment for most teams, but it has to be evaluated per task on a blind sample. Global model downgrades without per-task evaluation are how products quietly get worse.
How much of the bill should inference be?
There is no universal figure, but it should be a stable or falling share of revenue as you grow. A rising share is the signal to act, regardless of the absolute number.
Should we self-host to save money?
Only at sustained high volume, and only after counting engineering time, hardware utilisation and on-call. Below that threshold, the saving is usually smaller than the operational cost of owning it.
What is the fastest saving available?
Usually prompt trimming plus caching on the highest-volume endpoint. Both are days of work, carry little quality risk, and are measurable within a week.