Tools & Buying
How to Read AI Product Claims Without Being Fooled
By Jim Vernon, Editor, AI Intelligence International · Published 13 February 2026 · Reviewed against our editorial standards · About the author
Vendor claims are not usually false. They are true under a definition you were not shown, measured on a dataset you cannot see, against a baseline nobody would choose.
Six questions turn a marketing number back into information.
Key takeaways
- Accuracy compared to what?: An accuracy figure needs a baseline and a task definition.
- Time saved by whom: Time-saved claims typically measure the drafting step in isolation and ignore review, correction and rework.
- Benchmark wins: Public benchmarks measure narrow capabilities and are frequently contaminated by training data.
- Case studies and social proof: Read for specifics: named person, named process, before-and-after numbers with a method.
Accuracy compared to what?
An accuracy figure needs a baseline and a task definition. Ninety-five percent accurate at what, measured against whose judgement, on which distribution of cases?
Ask for the error breakdown. A system with a low overall error rate concentrated in your highest-value cases is worse than a mediocre one that fails uniformly.
Insist on evaluation with your data before believing any number.
Time saved by whom
Time-saved claims typically measure the drafting step in isolation and ignore review, correction and rework.
Ask for end-to-end cycle time from an existing customer, not step-level savings from a lab study.
If the vendor cannot name a customer who will talk to you, treat the figure as illustrative.
Benchmark wins
Public benchmarks measure narrow capabilities and are frequently contaminated by training data. A benchmark win predicts very little about your workflow.
Ask which benchmark, when, against which competitor versions. The comparison is often against an older release.
Your twenty-case evaluation set is worth more than every leaderboard combined.
Case studies and social proof
Read for specifics: named person, named process, before-and-after numbers with a method. Case studies without these are testimonials.
Ask what did not work. Vendors with mature deployments can answer this; vendors with pilots cannot.
Ask for a reference customer of your size and sector, and speak to them without the vendor present.
Six questions to send before a demo
What exactly does the accuracy figure measure and against what baseline? Which model version, and what is your policy when it changes? Where is data processed and is it used for training? What does a customer of our size spend in year one including implementation? Can we pilot with our own edge cases? What is the most common reason customers churn?
The last question is the most revealing, and the willingness to answer it tells you more about the vendor than the answer itself.
Translating three common claims
'Ninety-nine per cent accurate' usually means on a vendor-curated set, on a task narrower than yours, scored by the vendor. The question that unpicks it is simple: accurate at what, against which baseline, measured by whom.
'Saves ten hours a week' is typically the best result from a friendly pilot, extrapolated. Ask for the median across all pilot users and the number of users who saved nothing; the second number is more informative than the first.
'Enterprise-grade security' is a marketing phrase, not a certification. Ask which standard, which audit report, its date, and which subprocessors handle your data.
Reading a case study properly
Look for the baseline. A case study without a stated before-state is a testimonial, and testimonials tell you the customer was pleased, not that the tool caused the outcome.
Check whether the named contact still works there and whether the deployment is still running. Vendors rarely retire case studies when customers churn, and a two-year-old success story is not evidence about the current product.
Ask for a reference customer of similar size in a similar sector. A refusal is informative; an enthusiastic introduction to a company twice your size in another industry is only mildly useful.
What to demand in the trial agreement
Insist on trialling with your own data, your own users and a defined success measure written before the trial starts. Demos run on the vendor's data prove only that the vendor prepared well.
Set an exit condition in the same document: if the measure is not met by the end date, the trial ends without a renewal conversation. Trials without exit conditions become subscriptions by inertia, which is precisely what the claims were designed to achieve.
Keep a record of what was promised
Copy the specific claims that influenced your purchase into your own notes with the date and source page. Vendors update marketing pages, and a promised capability can quietly become a roadmap item.
At renewal, check the list. Claims that never materialised are legitimate negotiating material and, at minimum, they calibrate how you read the next round of announcements.
This habit takes five minutes per purchase and is the cheapest protection available against buying the same optimism twice.
Frequently asked questions
Are vendor demos ever useful?
For understanding the interface and workflow, yes. For predicting performance on your data, no — always pilot.
How do I evaluate a startup versus an incumbent?
Startups on fit and responsiveness, incumbents on governance and continuity. Score both dimensions explicitly rather than by instinct.
Is a free pilot a good sign?
Only if it uses your data and your edge cases. A free pilot on curated sample data is a demo with extra steps.
What claim should always trigger scepticism?
Any percentage without a stated baseline, task and dataset — and any promise of full automation of a judgement-heavy process.
Are third-party benchmarks trustworthy?
More than vendor ones, but they still measure generic tasks. Treat them as a shortlist filter, never as the final decision for your specific workflow.
What is the single best question to ask a vendor?
'Show me a case where your product performed badly and what you changed.' Honest vendors answer it well; the rest change the subject.