How to Review an Agent's Work When You Can't Read All of It

Reviewing a chatbot's answer is easy: there's one output, you read it, you decide.

Reviewing an agent's work is a different problem, and most people haven't noticed that it's different. An agent that took forty steps to produce a result generates a trace — searches run, files opened, calls made, intermediate conclusions drawn — that is longer than the thing you asked for. Nobody reads it. So the practical review becomes does the final answer look plausible, which is the same review you'd give a chatbot, applied to something with forty places to have gone wrong.

The answer isn't reading more. It's reading a different part.

The final output is the least informative part

A wrong answer produced by a sound process is usually obvious. A wrong answer produced by a broken process is usually not, because the model will have written a confident summary of whatever it found — including if what it found was nothing, or the wrong thing.

This is the specific failure that surprises people. The output reads well because writing well is what the model is best at. The error is three steps back, in a search that returned nothing useful and got summarised as though it had.

So the question to bring to a trace isn't "is this right?" It's "where did the information in this actually come from?"

The four checks that cover most of it

Check the inputs, not the conclusions. Which sources did it actually open? An agent asked to summarise last quarter's support tickets, that read four of them, has produced a summary of four tickets. That fact is in the trace and absent from the output.

Check the point where it stopped looking. Most agent runs contain a moment where it had enough to proceed and stopped gathering. Find that moment and ask whether you'd have stopped there. This is where scope quietly shrinks.

Check anything irreversible separately. Sends, spends, deletes, writes to shared systems. These don't get sampled — they get read individually, every time, because the cost of being wrong isn't symmetrical with the cost of checking.

Check one claim at random, properly. Not the most important claim; a random one. Tracing a single arbitrary statement back to its source tells you more about the run's reliability than skim-reading the whole thing, because you can't fool yourself about whether the check passed.

That's typically five minutes on a run that took the agent twenty. It is not the same as reading everything, and it isn't meant to be.

What to do when the run is too long to sample

Sometimes the honest answer is that the task was too big to supervise. An agent given six hours of work and full autonomy hasn't produced something reviewable — it's produced something you either accept or discard.

The fix is upstream: shorter runs with a checkpoint between them. Three runs of forty minutes with a look in between is not slower than one run of two hours, because the two-hour version includes the time you spend either reviewing it properly or worrying that you didn't.

Set the threshold before you look

The thing that makes review fast is deciding, in advance, what would make you reject the work. Two or three specific properties: every figure traceable to a named source, no claim about status that wasn't in the source data, nothing sent externally without a read.

Written down, review becomes comparison. Undecided, review becomes forming an opinion while looking at something designed to be persuasive — which is a much harder job, and one you'll get tired of doing within a fortnight.

Idris — the summary of four tickets

Idris runs a support function at a logistics company. He'd set an agent to produce a weekly themes report from the previous week's tickets, and for a month he read the report and found it reasonable.

The week he opened the trace, he found it had been sampling. It read a subset, wrote a confident summary, and never said which subset or how large — because he'd never asked it to, and it had no reason to volunteer that the sample was twelve tickets out of four hundred.

The reports hadn't been wrong exactly. They'd been a summary of a small unrepresentative sample, presented in the register of a summary of everything.

Hala — the claim she picked at random

Hala is a policy analyst in local government. Her habit on agent-produced briefings is to pick one factual claim at random and trace it back properly — not the headline claim, a boring one.

Her reasoning: the headline claim gets checked by everyone, so it's the one the run is least likely to be wrong about. A random claim tells her what the process is like when nobody is watching it.

Over six months this has caught three things, all of the same shape: a figure that was real, from a real source, about a slightly different thing than the briefing said it was about.

The one thing

Reviewing an agent means reading the middle, not the end. The output is the part the model is best at producing and the least informative about whether the work was sound.

Check what it read, where it stopped looking, everything irreversible, and one claim at random. If a run is too long for that to be possible, the run is too long.