Rollback is not redeploying old code. It is not copying a single file or reverting a commit — it is undoing a deployment where the bad version has already had time to act: spending money, sending messages, writing data, or issuing credentials under the wrong rules. A developer once documented a runaway agent loop burning over $700 in 72 hours with no budget controls. That is not a redeploy; that is a crisis that needs a structured recovery. Practitioners on Hacker News frame the core question as who can use which agent, and what each agent can access — and a rollback is where you find out whether you can actually answer it for the version you are trying to remove.
The Governed Agent Mesh Playbook treats every deployment as a plan bound: a complete statement of the agent's identity, model, prompt, tools, data scope, time window, egress rules, storage limits, cost ceiling, expiry, and grant references. Rolling back means restoring the whole bundle and appending an honest record of the recovery to the audit trail — not erasing the bad version from it. This AI agent rollback checklist walks that recovery in order.
What counts as the version you are rolling back¶
A bad deployment is not just a misconfigured script. It is a bundle of interdependent components: the model, the prompt, the tools, the agent identity in the format agent:[OWNER]/[NAME], the data class it may touch, the resources it may use, the time window it operates in, the domains it may send data to and how much it may transfer, the storage scope, the cost ceiling, the expiry, and the grants it holds. If any of those are missing from the version you restore, you have not undone the bad version — you have left part of it in place and renamed the result.
Consider the common shortcut: the rollback restores the old prompt and model, but the bad deployment also widened a grant, and the grant stays widened. Nothing looks broken — the agent simply keeps access it was never supposed to have, under a version everyone believes is safe. A rollback has to be atomic across the bundle. And if a bound was never recorded in the first place, that absence is itself a finding: a unit of work whose bounds cannot be stated was never a plan.
The AI agent rollback checklist¶
- Freeze first. Before you touch anything, stop the agent from doing new work. Use the kill switch or a fail-closed block so new requests are blocked and recorded, and park in-flight work instead of letting it finish under the bad rules. An agent that is still acting is still spending, writing, and sending. Record the freeze itself as an audit row — who froze it, when, and why — so the record proves the harm stopped before the repair started.
- Snapshot the bad state. Preserve the bad state exactly as it stands before you change a single setting: the model, prompt, tools, plan bound, and grants in force, plus the hash-linked audit rows written under them. Export or copy them in a form that cannot be quietly altered later. Never edit, delete, or renumber audit rows to make the incident look cleaner — the rows are evidence, not a draft.
- Name the last known-good bundle. Identify the last known-good bundle by name and version, and get its owner to approve the rollback with the reason recorded. A rollback nobody approved is just another unplanned change. If the recovery tempts you to raise a spend ceiling or widen a grant to make it work, treat that as a hard-gated action needing human approval and a recorded reason — never a shortcut taken mid-incident.
- Restore the plan bound and grants together. Restore the whole bundle at once: model, prompt, tools, the agent identity in the format
agent:[OWNER]/[NAME], data class, resources, time window, egress allowed domains and transfer cap, storage scope, cost ceiling, expiry, and every grant the plan references. Old code running with the bad version's widened grants is not a rollback — it is the same exposure wearing the old version's name. This is where most rollbacks quietly fail: the visible artifact is restored, the invisible permissions are not. - Revoke credentials issued under the bad version. Any credential, token, or grant minted while the bad version was live belongs to that version's judgement, not to the restored one. Revoke them, then reissue least-privilege access tied to the restored plan bound, with its own expiry. If you cannot say which identity holds a credential, which grant issued it, and when it expires, it is a standing permission with no owner — and a backdoor the rollback leaves open behind it.
- Account for side effects one by one. List everything the bad version did in the world, then work the list. Idempotent writes made under an idempotency key can usually be reversed or safely replayed against the restored plan. External effects cannot: a sent email, a submitted form, a payment, or a message to a customer cannot be unsent, and pretending otherwise corrupts the record. For each irreversible effect, record what happened, who it affected, and what disclosure or correction the owner owes. Reversible where possible, honest where not — that is the whole rule.
- Verify the audit chain. After the rollback entries are appended, verify the chain still verifies end to end: fetch the rows in sequence order, confirm each row's previous-hash matches the prior row's hash, recompute the digest, and confirm the sequence increments with no gaps — including across the freeze, the restore, and the re-enable. The rollback must extend the chain, never rewrite it. Any failure at any step fails the whole chain: investigate it, do not skip it, and do not declare the rollback done on top of evidence you have not checked.
- Re-enable in stages. Do not flip the restored agent straight back to full production. Start draft-only or in shadow, where outputs are recorded but nothing leaves. Then move to bounded-auto under a low per-plan spend ceiling that fails closed when reached. Only after staged runs match the known-good bundle's behaviour do you restore normal tiers — and raising a ceiling back is a hard-gated approval with a recorded reason, not an autopilot final step.
- Run a post-rollback review. While the details are fresh, record what changed, what the bad version did, what the rollback reversed, and what it could not. Then turn the failure into a gate: add the specific miss — the untested prompt, the widened grant, the missing bound — to your pre-deployment checklist so the next deployment is blocked by it before production, not after. A rollback that teaches the gate nothing will be run again.
What breaks when rollback is improvised¶
Improvised rollback fails in predictable places. Without a recorded known-good bundle, the team guesses at what “good” was — and guesses become the new configuration, unaudited. When history is rewritten to remove the bad version's rows, the audit trail stops being evidence and the next incident starts from a log nobody can trust. When widened grants are left in place, the exposure survives the recovery. When side effects are ignored, customers and counterparties learn about the incident from its effects instead of from you.
The quietest failure is the retry. Blocked actions are never retried automatically, and missing evidence is never degraded to a warning so work can continue — that fail-closed rule keeps a bad version from re-running its worst actions during cleanup. An improvised rollback that helpfully replays the queue or re-runs parked work can do more damage than the deployment it was meant to undo. Freeze means frozen until the restored bundle says otherwise.
Before you call it rolled back¶
- The bad version is frozen, and the freeze is recorded as an audit row.
- The bad state — bundle and audit rows — is snapshotted, unedited.
- The last known-good bundle is named, and an owner approved the rollback with a recorded reason.
- The full plan bound and its grants are restored together, under the same agent identity.
- Credentials and grants issued under the bad version are revoked and reissued at least privilege.
- Every side effect is listed: reversed where reversible, disclosed where not.
- The audit chain verifies end to end, across the rollback rows themselves.
- The agent is re-enabled in stages, under a spend ceiling that fails closed.
- The post-rollback review is written, and the failure is now a pre-deployment gate.
If any line on that list is missing, the rollback is not complete — it is a pause with good intentions. The Governed Agent Mesh Playbook — Studio Edition gives you the full system behind this checklist: the six policy templates, the plan-bound and audit-row schemas, the risk-tier worksheet, and the 14-day rollout that installs identity, grants, audit, spend limits, and approval gates in order. Put the bundle in place before the next bad deployment, and rollback becomes a procedure you run — not a crisis you improvise.