Tools & Buying
How to Choose Between AI Models Without Reading Benchmarks
By Jim Vernon, Editor, AI Intelligence International · Published 11 January 2026 · Reviewed against our editorial standards · About the author
Model leaderboards move monthly and correlate weakly with day-to-day usefulness. A model that wins on reasoning benchmarks can be the wrong choice for summarising your support tickets.
The reliable method is boring: build a small evaluation set from your own work and run the candidates against it. It takes an afternoon and outperforms any amount of benchmark reading.
Key takeaways
- Build a twenty-case evaluation set: Take twenty real tasks you would actually delegate: ten typical, five hard, five unusual.
- Score on the dimensions you care about: Usually: factual reliability, instruction adherence, formatting discipline, tone, latency and cost.
- Cost is a design variable: Price differences between tiers are large, and the cheapest capable model is usually the right production choice.
- Non-model factors that decide it: Data handling terms, region of processing, availability guarantees, rate limits, and how quickly the vendor deprecates versions.
Build a twenty-case evaluation set
Take twenty real tasks you would actually delegate: ten typical, five hard, five unusual. Write down what a good answer contains for each.
Twenty is enough to see clear differences and small enough to re-run when a new model appears. Store it where you will find it again.
Do not use public test questions. Everyone tunes on those, and your work does not resemble them.
Score on the dimensions you care about
Usually: factual reliability, instruction adherence, formatting discipline, tone, latency and cost. Weight them yourself — for bulk classification, cost and speed dominate; for client-facing drafts, adherence and tone do.
Score blind if you can. Knowing which model produced an answer biases scoring noticeably.
Record the results with dates. The comparison is only useful over time if you can see how it changed.
Cost is a design variable
Price differences between tiers are large, and the cheapest capable model is usually the right production choice. Reserve the expensive model for the small share of hard cases.
Routing simple tasks to a small model and escalating only on uncertainty is the standard pattern for controlling spend at volume.
Include the cost of long context: retrieved documents and lengthy system prompts can dominate the bill.
Non-model factors that decide it
Data handling terms, region of processing, availability guarantees, rate limits, and how quickly the vendor deprecates versions. These often matter more than a few points of accuracy.
Also consider ecosystem: whether the model is available inside the tools you already use, which removes an integration project.
Avoid designs that assume one specific model forever. Keep the prompt and evaluation layer portable.
Re-running the decision
Re-evaluate twice a year, or when a vendor announces a change affecting your tier. Do not chase every release; switching costs are real and the gains are usually marginal for a well-tuned workflow.
Keep the evaluation set current by adding cases that failed in production. Over time it becomes the most valuable artefact in your AI programme, and it is the thing that makes every future selection decision cheap.
A worked example: choosing a model for support replies
A twelve-person software company needed draft replies for inbound support email. They pulled twenty real tickets: ten routine password and billing questions, five that referenced an unusual configuration, and five where the customer was angry and partially wrong about what had happened.
Three candidate models produced drafts for each ticket. The frontier model wrote the best prose but invented a refund policy twice. The mid-tier model was blander and never invented policy. The small model was fine on the ten routine tickets and unusable on the rest.
The decision was not 'which model is best' but 'which model plus which guardrails'. They shipped the mid-tier model with the refund policy pasted into the system prompt, and routed anything containing the word refund to a human. Cost fell by roughly two thirds against their first instinct.
Failure modes matter more than average scores
Two models can score identically on your evaluation set and fail in completely different ways. One might be slightly wrong often; the other exactly right most of the time and catastrophically wrong occasionally. For customer-facing work, the second profile is usually worse even when the average is better.
Record the shape of each failure, not just a pass or fail mark. Note whether the model fabricated a fact, ignored a formatting instruction, drifted in tone, or refused a reasonable request. Those four categories have different fixes and only one of them is solved by switching models.
Fabrication is addressed by supplying source material and asking for citations. Instruction drift is addressed by shorter, ordered instructions. Tone is addressed with examples. Over-refusal is usually a prompt framing issue. Switching models to fix a prompt problem simply moves the problem.
Write the decision down before you forget why
Six months later nobody remembers why the current model was chosen, so the decision gets relitigated from scratch every time someone reads a launch announcement. A one-page record prevents that: date, candidates, evaluation set location, scores, the decision, and the two or three constraints that drove it.
Include the conditions that would change your mind — a price cut past a threshold, a data-residency option becoming available, or a persistent failure category appearing in production. That turns a vague sense of unease into a trigger you can actually check.
Share the page with whoever pays the bill. Finance rarely disputes a model choice that arrives with an evaluation attached, and the same document answers the security team's first three questions without another meeting.
Frequently asked questions
Are open-weight models worth considering?
Yes where data residency, cost at scale or customisation matter, provided you can operate them. Include hosting and maintenance in the comparison.
How often do rankings change?
Frequently enough that chasing them is wasteful. Your own evaluation set changes far more slowly and is what should drive decisions.
Should I use several models?
Routing by task complexity is common and cost-effective. Using several arbitrarily creates maintenance and evaluation overhead with no gain.
Does a bigger context window help?
Only if your task needs it. Long contexts raise cost and can dilute attention to the instructions that matter.
How long should the evaluation take?
An afternoon for the first run. Once the twenty cases are written and scored, re-running them against a new candidate is usually under an hour.
What if two models tie?
Pick the cheaper one, or the one already available in the tools your team uses. A tie means the difference is not worth an integration project.