Unit 9 of 9 · Intermediate

Where agents fail (honestly)

In one read

Before trusting an agent, an executive should understand exactly how they fail — and the honesty here is the point. The first mode is hallucination: an agent can be confident, fluent, and wrong, stating something false in the same assured tone it uses for the truth. The second is drift: over a long run it slowly wanders off the original task, each step reasonable, the destination not. The third is overconfidence paired with silent errors — failures that look like success, where nothing raises an alarm and the wrong result simply flows onward. These are not bugs waiting for a patch. They are properties of the technology, to be bounded rather than eliminated: hard limits on what the agent may do, human sign-off where the stakes are real, and verifiable evidence of what actually happened instead of the agent's own word. Grasping this is the honest foundation for trusting one at all.

The honest part

Every unit so far described what agents can do. This one is about how they fail — and the honesty is not a caveat, it is the lesson. You cannot bound what you refuse to name, and an executive deciding whether to trust an agent needs these named plainly.

Three failure modes to understand

  • Hallucination — confident, fluent, wrong. An agent can state something false in exactly the same assured tone it uses for the truth. The fluency is not evidence of correctness; a smooth, well-phrased answer can be entirely made up. This is the failure people underestimate most, precisely because it sounds right.
  • Drift — slowly wandering off task. Over a long run, an agent can ease away from the original objective. No single step looks wrong; each is locally reasonable. But the accumulated path arrives somewhere you never asked for. The longer and more autonomous the run, the more drift matters.
  • Overconfidence and silent errors — failures that look like success. The most dangerous errors are the ones that raise no alarm. The agent reports done, the output looks plausible, and a wrong result simply flows onward into the next step. Nothing crashed; nothing flagged; the mistake is invisible until it is expensive.

These are properties, not bugs

The instinct is to treat these as defects that a better model will patch away. That is the wrong frame. They are properties of how the technology works, and the honest posture is to bound them rather than pretend they are gone:

  • Limits — hard ceilings on what the agent may do, so a failure is contained.
  • Human sign-off — a person in the loop exactly where the stakes turn real and irreversible.
  • Verifiable evidence — a record of what actually happened that you can check yourself, rather than trusting the agent's own account of it.

Those three — bounding authority, sign-off at the stakes, and evidence you can verify — are the foundation the rest of agent trust is built on. Everything that follows in serious agent systems is a way of making those concrete.

See it in kestrel

Kestrel does not claim these failure modes are solved. Its answer is the third item above made literal: every run produces a record you can recompute yourself, so you check the math rather than take the agent's word. See one on a generic instrument, no signup, no card:

npx kestrel.markets sim fomc-rate-decision-whipsaw

It prints a certified proof URL you can recompute on your own machine, byte for byte — the evidence, not a promise.

Recompute it

Every claim in this unit recomputes from a certified proof — no account, no card.

/proof/art_d29415f0cf502f4a218a9cba

Agent twin: /education/agents-fundamentals/af-where-agents-fail/content.md