Why 'It Looks Right' Isn't a Verification Check

If your review process for AI output is "read it and see if it looks right," you're running the one check that AI is specifically built to pass. Four separate incidents this year prove it in four different formats — citations, a second AI's fact-check, code, and an agent's finished deliverable — and each one needed a different fix, not a more careful read.

Four failures, one shape

In May, 102 of 148 citations in a Cabinet-approved South African policy paper turned out to be fabricated. The document had been formally approved and circulated before anyone opened a reference. It read like a properly sourced paper because AI is good at producing things that read like properly sourced papers.

In June, research on frontier models fact-checking each other found 67% disagreement on real-world claims — 77% in legal contexts, 71% in health. Asking a second AI to confirm the first isn't a check. Both models are running the same kind of pattern-matching over the same kind of text, so they agree on the obvious cases and split on exactly the ones that matter.

That same month, an AI agent got code merged into a production release of Fedora Linux because the patches were coherent, plausible, and wrong. Human reviewers read the diffs, found them reasonable, and approved them. Nobody ran the fix against the bug it claimed to close.

In July, OpenAI's own safety guidance for its new agent product admitted the system produces "finished mistakes at a scale that is harder to catch" — output that's polished, complete, and wrong in a way that a glance at the finished product won't surface.

Citations, a second model, code, a finished deliverable. Different artifacts, same failure: everyone involved checked whether the output looked like what a correct answer would look like, and every one of those outputs was built to clear exactly that bar.

The fix is matching the check to the claim

"Looks right" isn't a weak check you can strengthen by reading more slowly. It's the wrong check, because it tests fluency and AI fluency is not correlated with accuracy. The check that actually works depends on what the output is claiming:

A citation, quote, or statistic — open the source and confirm the specific claim appears in it. Not that the source exists. That it says what the AI said it says.

A second AI's review of the first AI's work — doesn't count as verification on its own. Use it to generate where to look, then check that specific spot against a primary source yourself.

Code, a calculation, an agent's action — run it against the actual requirement. A diff that reads cleanly tells you nothing about whether the bug is gone; only running the fix does.

A finished-looking deliverable from a long agent run — decide in advance which two or three claims in it would hurt you if wrong, and check those specifically. "It looks done" is not a review; naming what you're going to verify before you look is.

Naomi: the two-model habit that wasn't a check

Naomi runs competitive research at a 70-person B2B software company. When a market-sizing figure in a quarterly briefing turned out to be two years stale, she added a step: run every summary through a second AI tool and flag disagreements before the briefing goes to leadership.

It caught nothing for three cycles, which she read as the process working. After the 67% disagreement research, she ran a test — she fed the same source document to both tools with a deliberately outdated figure planted in her prompt. Neither flagged it. Both models were reasoning from the same kind of surface pattern, so neither had a way to catch what the other missed. She now spends the same ten minutes differently: for the two or three figures that actually go in front of leadership, she opens the source document herself instead of asking a second model to bless the first one's summary.

Grant: reviewing the diff instead of the fix

Grant leads a four-person engineering team at an early-stage logistics startup. His team uses an AI coding agent to draft fixes for reported bugs, and his review step was reading the diff: does the change look targeted, does it match the ticket, does it avoid touching unrelated files. Clean diffs got merged same-day.

One fix looked exactly right — small, scoped to the right file, a commit message that matched the ticket — and shipped. The underlying bug reappeared two weeks later; the patch had handled the specific input in the bug report but not the general case. Grant's team now requires every agent-drafted fix to run against the original failing test case before merge, diff review or not. The diff tells you the change is plausible. Only the test tells you the bug is actually gone.

The one thing

None of these four failures involved careless people. A cabinet office, a research team, Linux maintainers, and OpenAI's own safety team all read their AI output and found it convincing — because convincing is what AI output is optimized to be. The check that catches fabrication was never going to be reading more carefully. It's picking, before you look, what would actually prove the claim — and going there instead.