Every incident review ends the same way. An autonomous agent does something irreversible, someone pulls up the logs, and within ten minutes the room has settled on a cause: the model hallucinated. We need tighter guardrails. Maybe a bigger model next release.
I want to offer a less comfortable explanation, because I have seen this exact failure before, and it had nothing to do with AI.
Key Takeaways
- AI agents are unreliable actors, not autonomous authorities.
- Traditional distributed systems patterns remain essential for agentic AI.
- Idempotency, sagas, reconciliation, and deterministic gates reduce operational risk.
- Trust is built through engineering, not prompting.
- Human oversight belongs at irreversible decision points—not every step of the workflow.
Long before today’s AI-agent boom, I was close to a payment service that charged a customer twice. No bug in the pricing, nothing anyone would call intelligence. The network dropped the acknowledgment after the charge went through, so the service assumed it had failed and tried again. Same card, two withdrawals. We did not fix it by making the service smarter. We gave every charge a unique key, so a repeated request was recognized and ignored. It never came back.
That is the story I keep returning to while the industry panics about agent reliability. The thing everyone fears — an autonomous process doing something irreversible, more than once, faster than a human can catch it-is not new. We solved it long ago. We just called it something else.
When “It Hallucinated” Is the Wrong Diagnosis
Blaming the model feels right: it is the interesting part, the piece we do not fully understand, so the piece we suspect. But look closely at the failures that actually cost money in 2026, and most are not failures of reasoning. They are failures of bookkeeping.
The agent commits the same seat to three customers because nothing stopped it from acting more than once. It reports a task as done when the side effect silently failed, and the next step builds on the lie. In most of these cases the model reasoned fine.
Why the 2026 panic over agent reliability is really distributed-systems amnesia
VEERA RAVINDRA DIVI
and where the answer has been sitting all along.
The system around it had no memory of what had already happened, no way to undo what went wrong, and no independent check on whether reality matched the agent’s account of it.
So why do we keep reaching for a better model? Because it asks the least of us. Building the machinery that makes an unreliable actor safe is unglamorous engineering, and it does not demo well.
“The model usually reasons fine. The system around it has no memory of what it did, no way to undo what went wrong, and no independent check on its own account of reality”.
An Agent Is Just Another Unreliable Node
Here is the reframe, straight out of distributed systems: an LLM agent is a non-deterministic actor that takes actions with side effects. That is the whole technical description. And distributed systems have been full of unreliable actors for decades — processes that die mid-write, networks that drop or duplicate messages, services that revive and finish a job everyone assumed was dead.
“An AI agent should be treated like any other unreliable distributed service: assume failure, design for recovery.”
We never made those actors reliable. That was never the point. We built systems in which unreliable actors could not do damage. A trading floor never trusted a single process to be correct. The agent is the newest member of that club — louder, more articulate, occasionally brilliant, and just as unable to promise it did a thing exactly once.
Once you see the agent as one more unreliable node, the fear stops being existential and becomes a checklist.
The Runbook Nobody Is Reading
The patterns that tame unreliable actors are the boring core of every serious transactional system, and they map almost one- to-one onto how agents hurt you.
The agent that acts twice is the idempotency problem. Attach a unique key to every action that changes the world, and a repeated request returns the first result instead of redoing the work. This is one of the key patterns Stripe uses to make payment retries safer. The key belongs in your tool layer, not your prompt. “Please don’t refund twice” is a hope; a key is a guarantee.
The agent that thinks it did something it did not is the dual-write problem, answered by the outbox pattern: record the intent to act in durable storage as part of the same change to your data, then let a separate process carry it out and mark it done. The agent is never trusted about whether the email sent — the record knows, even when the agent’s memory does not.
The long workflow that collapses under its own length is what the saga pattern was built for: break the job into steps, and give each step an action that undoes it. When step fourteen fails, you do not leave a half-booked, half-charged mess. You walk the undos backward to a clean state. Agents need sagas badly, and almost none have one.
“Trust isn’t created by better prompts. It’s built through idempotency, durability, reconciliation, and controlled execution.”
Then the pattern I see least, which matters most: reconciliation. You will not catch everything at the moment of action. That is why every trading system I touched ran an independent process that compared what it believed it had done against ground truth, and flagged every gap. Your agent’s memory of its own work is a claim, not a fact. Reconcile the claim.None of this requires understanding transformers. It requires treating a confident actor as still an unreliable one.
The One Problem That Actually Is New
Agents do add one thing the old runbook never faced. A payment service can only call the functions you wrote; an agent may propose unsupported actions or use available tools in ways you never anticipated, because reasoning toward a goal is why it exists. That open- endedness is the novel risk, and it is why the boundary now matters more.
The discipline is simple to state and hard to skip: the agent proposes, a deterministic gate disposes. Every irreversible action — money leaving, inventory committing, a message reaching a real customer — passes through a boundary you control, and anything outside it stops and asks a human. In my research I call this human–assured interoperability — not a person babysitting every step, but a person standing at the exact seam where a wrong move cannot be taken back. One gate in the right place beats a hundred lines of careful prompting.
The agent proposes. A deterministic gate disposes. Everything irreversible passes through a boundary you control — and a human stands at the seam.
What I Would Check Before You Ship
Take your riskiest agent and walk it through five questions. Not one is about the model.
- Idempotency: if any action fires twice, does the second do damage, or get quietly ignored?
- Durability: is the intent to act recorded before the action happens, so a crash mid-step is recoverable?
- Reversal: for every step that changes the world, is there a defined way to undo it?
- Reconciliation: is there an independent check comparing what the agent thinks it did against what actually happened?
- The boundary: does every irreversible action pass through a gate you control, with a human at the seam?
Wherever the answer is not clean, you have found your next sprint — worth more than another benchmark point.
Closing Thought
The teams shipping agents into real production this year are not the ones with the cleverest prompts or the newest model. They are the ones who felt a strange déjà vu, went back to the patterns that made payments and trading trustworthy long before any of this, and reached for the ledger instead of the model.
Intelligence was never the hard part. Trust was. And trust, in software, has always come from the same place: not from an actor you believe, but from a system built so you never have to.
