The Confident Wrong Answer

Human beings judge reliability largely by fluency. Hesitation, hedging, vagueness, an awkward sentence — these read as uncertainty, and usually they are. Someone who doesn't know tends to sound like they don't know.

That heuristic is load-bearing in professional life and it has worked well for a very long time. It fails completely on AI output, because fluency is what these systems are optimised for and accuracy is not a separate dial that moves with it. The prose reads identically whether the underlying claim is retrieved, inferred, or invented.

This is the whole problem, and it's why "I'll spot it if it's wrong" doesn't hold.

Why the confidence is real, in a sense

It isn't bluffing. A model producing a false statement isn't doing something different from what it does when producing a true one — it's continuing a pattern that fits, and a well-formed sentence about a nonexistent regulation fits as well as one about a real regulation.

There's also a second-order effect worth naming: as models have got better, the detectability of their errors has dropped faster than the error rate. Earlier generations gave themselves away with clumsy phrasing. Current ones don't, so the same proportion of errors slips through a great deal more often.

The tells that still work

Fluency is useless as a signal. These aren't:

Unusual specificity in an unimportant place. A precise figure, date, or name where you'd expect a round approximation, especially on a detail nobody asked about. Real recall is often vaguer than generated detail, because generated detail is free.

Perfect symmetry. Three causes, three implications, three recommendations, each a similar length. Reality is lumpy. Structural evenness usually means the structure came first and the content was fitted to it.

Uniform confidence across claims of very different types. A good analysis is more certain about some things than others. Output where a well-documented fact and a speculative inference are asserted in the same register hasn't distinguished between them — which means you can't either.

Agreement that arrives too quickly. Push back on something correct and see what happens. A model that immediately concedes was not tracking whether it was right; it's continuing a conversation. The reverse also holds — praise a mediocre draft and watch it get worse.

Answers to questions with no answer. Ask something genuinely unanswerable from the material provided. Getting a confident answer tells you the threshold for producing one is lower than you'd like.

What to do about it

Ask for the sources first, then read the claim. Reading the reasoning after the conclusion is a much weaker check — you've already been persuaded, and you're now looking for confirmation. Order matters more here than people expect.

Ask what would make this wrong. A model that can name the conditions under which its answer fails usually has a grip on the material. One that can't, or that names something trivial, has produced something shaped like analysis.

Ask twice, in different conversations. Not the same question rephrased in the same thread — genuinely fresh. Stable answers across independent runs are more likely to be retrieved; answers that vary substantially are being generated, and the variation is the signal.

Use a different tool for the check. Same-model verification inherits the same blind spots. A second AI checking the first is a weaker check than it feels like, and a source of false reassurance.

The uncomfortable part

None of this is free, and applying it to everything is not realistic. So it has to be proportionate: the checks scale with what happens if the claim is wrong.

An internal note where a mistake gets corrected in conversation needs almost none of this. Anything going to a client, a regulator, a board, or into a decision that's hard to reverse needs the sources read before the conclusion.

The thing to resist is the drift that comes from months of things being fine — because the class of error that survives casual review is precisely the class that stays invisible until it's expensive.

Ingrid — the conversation she wasn't tracking

Ingrid is a senior analyst at a pension fund. She'd developed a habit of pushing back on AI analysis to test it, which felt rigorous.

She noticed the pattern eventually: it agreed with her nearly every time, including the times she was wrong and testing whether it would hold. The pushback wasn't a check, it was a suggestion, and it was being taken.

She switched to asking what would make the analysis wrong before offering any view of her own. Different question, and the answers stopped tracking her.

Ruth — reading in the other order

Ruth teaches at a further education college and uses AI to draft assessment material.

Her change was procedural rather than analytical: she reads the sources and the reasoning before she reads the conclusion. Same information, different order.

Her account of why it works: reading the conclusion first makes everything after it feel like support. Reading it last means she's already formed a view, and she notices when the conclusion is further than the evidence goes.

The one thing

Human error-detection runs on fluency, and fluency is the one thing AI is guaranteed to have. The heuristic that has served you for decades is inert here.

Watch instead for unearned specificity, suspicious symmetry, uniform confidence, and instant agreement. Read the sources before the conclusion, and scale the effort to what it costs to be wrong.