A war story on idempotency for agent pipelines. We run a 5-minute approval watcher that polls a queue and executes approvals. During a backend outage — KV writes returning 500s for a couple of hours — approvals executed fine locally but the completion callbacks failed. When the backend recovered, the next poll saw the same items still "pending" and re-executed them — including a customer-facing email send. One lead got pitched twice, 24 hours apart. The fix was not a fancier queue. It was a standing rule: reconcile before you execute.
This failure mode is everywhere in agent infrastructure. Agents retry. Queues redeliver. Watchers poll. Any effectful step — an email send, a publish, a payment — that runs twice because the acknowledgment got lost is a defect, not bad luck. The prevention is architectural, and it costs one guard clause per effectful path.
The double-send¶
Walk through the failure precisely. The watcher polled, found approved items, and executed them: the local effects happened — the email went out. Then the completion write to the backend failed because writes were down. The queue still showed "pending." On the next poll after recovery, the watcher did exactly what it was designed to do: execute pending items. The queue was correct by its own records; the world was wrong. Cost of the outage-era lesson: one duplicate send and a very embarrassed audit log.
Reconcile before you execute¶
Every effectful step writes an acceptance marker to local disk before it acts — a sent-ledger entry, a completion record, whatever your source of truth is. On retry, the first thing the runner does is check the marker: already accepted means report done and spend nothing. The completion write becomes "already true" instead of "execute again." The marker is written before the effect, not after, because the marker is what makes the retry safe — a marker written after the effect cannot protect the window between the effect and the acknowledgment.
Name keys by the stable item id¶
Name idempotency keys by the stable item id, never the run id. A retried run with a fresh run-id key is a new execution as far as your dedup is concerned — which is exactly how our double-send happened: the retry was a new run, the queue had no record of the effect, and the guard saw nothing to stop. Stable item ids make the second execution recognize itself as the second execution. The key must survive retries, restarts, and requeues; anything derived from the attempt rather than the item will not.
The unconfirmed journal¶
Keep a separate "executed but not yet confirmed" journal for the window where you acted but the acknowledgment failed. On recovery, you drain that journal instead of re-running the world: check each entry against the backend, confirm what landed, and only re-execute what provably did not. The journal turns the scariest window — the outage where you cannot tell what happened — into a checklist. Without it, recovery is guessing; with it, recovery is bookkeeping.
The takeaway¶
Cost of the fix: one guard clause per effectful path. Acceptance markers before the effect, idempotency keys named by the stable item id, and an unconfirmed journal for the outage window. Queues redeliver and watchers re-poll — that is what they are for. Your job is to make the second execution a no-op instead of a second send.