An AI Agent's First Week: How to Supervise Without Babysitting

There are two ways people supervise AI agents, and both fail.

The first is approving everything, forever. It's safe and it doesn't scale — within a fortnight the approvals are being clicked without being read, which is worse than no checkpoint at all because it looks like one.

The second is deciding the agent seems reliable and switching it to autonomous. That's how the incident stories start, and they always start in week two or three, on the case nobody anticipated.

The workable answer is neither. It's a ramp, and the model that fits it best is onboarding a new colleague — where nobody expects either permanent supervision or unsupervised competence on day one.

Days one to three: you're not checking the work

The instinct is to review output for quality. That's not what this phase is for. In the first few days you're discovering the situations you failed to anticipate when you wrote the brief.

So read for surprise, not for correctness. Every time the agent does something you didn't expect — even something reasonable — that's a rule you didn't know you needed. Write it down. Most of them will be edge cases in your own process that nobody had ever articulated, because a human colleague would have handled them by asking.

Expect roughly five to ten of these. If you find none, either the task is genuinely simple or you're skim-reading.

The rest of week one: spot-check, don't review

Once the surprises stop arriving, switch to sampling. Check one in five, chosen at random rather than by which looks interesting.

Keep full approval on anything irreversible — send, spend, delete, grant — and drop it everywhere else. This is the step people skip, and skipping it is what makes supervision collapse into rubber-stamping: if the checkpoint applies to everything, it gets applied to nothing.

Week two onward: the permanent shape

Two things stay forever.

The irreversible categories keep their checkpoints. Not because the agent is untrustworthy — because the cost of being wrong is asymmetric. You are not checking whether it's right. You're taking responsibility for something that can't be taken back.

A periodic sample. Once a week or once a month, look at a handful of outputs properly. Agents drift when the inputs drift, and the failure is gradual — a categorisation that was right for last quarter's ticket mix and is subtly wrong for this one. Nothing alerts you to that.

What to do when it gets something wrong

The instinct is to add a rule about the specific error. Resist it once and ask a different question first: is this a rule I forgot, or a task the agent shouldn't have?

Sometimes the answer is genuinely the second one. An agent consistently mishandling a category of case is telling you that category needs a human, and the right fix is to route it out rather than to keep patching around it.

Priya — reading for surprise

Priya runs customer operations at a 120-person software company. She deployed an agent to triage inbound support tickets by urgency and route them.

In the first three days she logged eleven surprises. Nine were trivial. Two mattered: the agent treated any ticket mentioning a deadline as urgent, which swept up routine feature questions; and it had no way to recognise that a particular enterprise customer's "low priority" is everyone else's "today".

Neither was a mistake in any obvious sense. Both were context that lived in her team's heads and had never been written anywhere. She'd have found neither by reviewing for correctness, because each individual decision looked defensible.

Callum — the drift nobody noticed

Callum manages an operations team at a logistics firm. An agent had been categorising supplier exceptions accurately for four months, so the weekly sample had quietly lapsed.

In month five the company added a new carrier with a different exception vocabulary. The agent kept categorising confidently, using the closest match from the old scheme. Roughly a fifth of exceptions were landing in the wrong queue.

Nobody noticed for six weeks, because the outputs looked exactly as they always had. What surfaced it was a queue owner asking why her volume had jumped.

His rule now: any change to what goes into an agent triggers a fresh sample, regardless of how long it's been running well. The agent didn't get worse. The world changed underneath it.

The one thing

Supervision isn't a level of trust. It's a shape, and it changes over time.

Read for surprise in the first few days, sample in the first week, keep permanent checkpoints only where actions can't be undone — and re-open the sample whenever the inputs change, because that's when a reliable agent quietly stops being one.