Did the Agent Actually Do It, or Just Say It Did?

Checking whether an AI's answer is right is a familiar problem. Checking whether an agent's action happened is a different one, and it catches people out because the reporting looks the same either way.

"I've updated the records and notified the team" is produced in exactly the same confident register whether both things happened, one did, or neither. The model isn't lying — it has, in a meaningful sense, limited insight into what actually landed. It issued a call, the call returned something, and it summarised the intent.

Why the gap exists

Three ordinary reasons, none of them exotic:

Partial success reported as success. Forty records to update, thirty-six succeed, four fail on a validation error. The summary says the records are updated, because from inside the run that's substantially what happened.

Silent permission failures. It lacked access to a system and the tool returned an empty result rather than an error. Empty looks a lot like "nothing to do", and gets summarised as done.

The action that was prepared but not sent. A draft created, not delivered. A change staged, not committed. This one is common with anything that has a two-step shape, and the intermediate state reads as completion.

The pattern across all three is the same: an agent reports on its intent and its immediate response, not on the state of the world afterwards.

The check: go and look at the thing

The reliable verification isn't in the agent's output at all. It's in the system the action was supposed to affect.

Did the email arrive in Sent, with the attachment? Does the record show the new value and today's timestamp? Is the file where it should be, containing what it should contain? Ten seconds, in a different window, on the system of record.

Nothing in the agent's own report substitutes for this, because the report and the failure come from the same place.

Ask for counts, not for confirmations

You can make the gap much easier to spot by changing what you ask for.

Tell me how many records you attempted, how many succeeded, and list any that failed.

A number is checkable in a way that "done" is not. It also changes what the model attends to — asked for a count, it has to look at results rather than summarise intentions. Where an agent can't produce the count, that's itself informative: it means it doesn't know, which means neither do you.

The same applies to anything with recipients, files, or rows. Ask what, how many, and which ones failed.

Watch the ones that end in a different system

The failure clusters in a specific place: actions that finish somewhere the agent can't read back. It can usually confirm what it wrote to a document it still has open. It usually cannot confirm that a message arrived, a payment cleared, or a downstream system accepted the record.

If an action ends outside the agent's reach, the confirmation has to come from outside too. That's the category worth a standing check, and it's a short list for most workflows — usually two or three actions.

Once, then periodically

This isn't a permanent tax. Verify a new action type properly the first several times, until you know its failure mode; then move to periodic sampling.

The reason to keep sampling rather than stopping is that these failures are usually environmental — a permission that expired, an API that changed, a mailbox that filled. They appear without anything about the agent changing, which means past reliability tells you nothing about this week.

Ondrej — the thirty-six of forty

Ondrej runs a membership organisation's operations. A renewal agent reported updating member records each morning and had done so, reliably, for weeks.

A member complaint led him to check the records directly. Around one in ten updates had been failing on a validation error introduced by an unrelated form change — and the agent's summary had said "records updated" every single time, because most of them had been.

His fix was the smallest possible one: ask for attempted, succeeded, and a list of failures. The number now appears in the daily report and a non-zero failure count is visible in a glance.

Nadia — the drafts that never sent

Nadia is a fundraising manager at a charity. Her outreach agent reported sending personalised follow-ups to a segment of lapsed donors.

They were all sitting in Drafts. The tool it used created and did not send, which was a configuration detail nobody had looked at, and the agent had summarised the successful creation as the completed action — accurately, from its own point of view.

Her standing habit now is that any action ending in another system gets checked in that system. It takes ten seconds and it has caught two further instances of the same shape.

The one thing

Agents report intent, not outcome, and the two diverge most on partial successes, silent permission failures, and actions that were prepared rather than completed.

Ask for counts instead of confirmations, and check anything that lands in another system in that system. It's ten seconds, and the agent's own report can't do it for you.