Business & Money
Measuring AI Output Quality Once It Is in Production
By Jim Vernon, Editor, AI Intelligence International · Published 7 February 2026 · Reviewed against our editorial standards · About the author
Most organisations evaluate carefully during a pilot and then stop. Production quality is left to complaints, which is the slowest and most expensive detection mechanism available.
A workable quality system has four parts: a written rubric, systematic sampling, drift detection, and a defined exception path. None of them require specialist tooling.
Key takeaways
- Write the rubric before you need it: A rubric separates blocking errors from preferences.
- Sample systematically: Pick a sampling rate you can sustain forever rather than an ambitious one you will abandon.
- Detect drift deliberately: Maintain a fixed evaluation set of real cases with known good answers, and re-run it monthly and after every vendor update.
- Design the exception path: Decide what happens when the system is uncertain or the case is unusual: who receives it, how quickly, and what they are authorised to do.
Write the rubric before you need it
A rubric separates blocking errors from preferences. Blocking errors are factual mistakes, policy breaches, wrong figures, missing disclosures, unsafe recommendations. Preferences are tone, structure, word choice.
Without this separation, review degenerates into taste and reviewers burn out arguing about phrasing while a wrong number goes out.
Keep it to one page with examples of each category. Examples do more work than definitions.
Sample systematically
Pick a sampling rate you can sustain forever rather than an ambitious one you will abandon. Even a small random sample per week, scored consistently, detects problems far earlier than complaints do.
Stratify the sample: some routine cases, some high-value, some unusual. Pure random sampling under-represents exactly the cases where errors are expensive.
Record scores over time in one place. The trend matters more than any single week's number.
Detect drift deliberately
Maintain a fixed evaluation set of real cases with known good answers, and re-run it monthly and after every vendor update. This is the only reliable way to notice that behaviour changed.
Include cases that previously failed and were fixed. Regression on old failures is common and embarrassing.
Keep the set small enough to re-run in an hour. A set that takes a day will not be re-run.
Design the exception path
Decide what happens when the system is uncertain or the case is unusual: who receives it, how quickly, and what they are authorised to do. Systems without an exception path fail by producing a confident wrong answer.
Make uncertainty visible in the interface. Reviewers calibrate much better when they can see which parts of an output are weakly supported.
Track exception volume as a health metric. A sudden fall usually means the routing broke, not that quality improved.
Fact-grounding for anything factual
Where output contains claims, figures or citations, require sources and verify a sample of them. Fabricated references remain the most damaging failure mode because they are plausible and quotable.
Use a fixed checklist for the categories that matter in your context — numbers, names, dates, legal claims, quotes — and require the reviewer to tick them explicitly rather than reading holistically.
Publishing an internal register of caught errors, blameless and specific, improves reviewer performance faster than any training.
Reporting it upward
Report three numbers monthly: blocking error rate in the sample, exception volume, and evaluation-set score. Together they say whether the system is safe, whether it is coping, and whether it has drifted.
This is also the evidence base you need for the next expansion decision, and for any regulator or auditor who asks how you supervise the system. Building it early is far cheaper than reconstructing it later.
Build the evaluation set before you need it
A frozen set of fifty real inputs with agreed-good outputs is the single most useful artefact in any AI deployment. It turns 'the model feels worse this week' into a number you can compare against last month, and it is the only way to safely change a model, a prompt, or a retrieval source.
Draw the cases from production, not imagination: thirty typical, ten awkward, ten that previously failed. Keep them fixed. The moment you edit the set to make results look better, you have lost the measurement.
Re-run it on every prompt change and every model change, and record the results with a date. Two data points a month is enough to spot drift that nobody would otherwise notice until a customer complained.
Pair automatic scores with a small human sample
Automatic scoring — exact matches, regex checks, or a second model grading against a rubric — is cheap and catches regressions fast. It is also confidently wrong about tone, usefulness and subtle fabrication, which are precisely the failures customers notice.
The workable combination is automatic scoring on every run plus ten human-reviewed outputs a week. The human sample is small enough to survive a busy month and large enough to catch a category of failure the automatic checks cannot see.
Log every production output with its inputs and a version tag for prompt and model. Without version tags, a quality investigation becomes archaeology and usually ends with a guess.
Frequently asked questions
Who should do the sampling?
Experienced practitioners, rotating, with the rubric in front of them. Reviewers who never do the underlying work drift toward superficial scoring.
How do we score subjective outputs?
Score against the rubric's blocking categories only, and treat everything else as commentary. Chasing agreement on taste is not a useful use of review time.
Is automated evaluation worth it?
As a screen for obvious failures, yes. As a replacement for human sampling on high-stakes work, no — the failure modes correlate.
What error rate is acceptable?
Whatever your process tolerated before, measured the same way. The right benchmark is the human baseline, which is rarely as good as people assume.
Is model-graded evaluation trustworthy?
For relative comparisons and regression detection, largely yes. For deciding whether output is good enough to ship, keep a human in the loop.
What quality threshold should trigger a rollback?
Set it before launch — commonly any drop in the frozen-set pass rate beyond normal week-to-week noise, plus any new fabrication in customer-facing text.