Running AI agents in production introduces a non-deterministic failure mode that traditional SRE playbooks often struggle to categorize. Unlike a crashing microservice, an agent can remain "healthy" from an infrastructure perspective while systematically corrupting data or executing unauthorized actions via tool-use. To manage this, on-call teams need a severity rubric that focuses on the outcome of the agent's autonomy rather than the state of the server.
The Four-Level Severity Rubric
When triaging an agent incident, avoid debating the "why" and focus on the "what." Use these four dimensions—blast radius, data exposure, financial impact, and reversibility—to slot the incident into a severity level.
SEV-1: Critical / Total Compromise
- Blast Radius: Global impact; agent is malfunctioning for all users or has gained unauthorized elevated privileges across the environment.
- Data Exposure: PII or secrets are being leaked in prompts or logs to unauthorized parties; wide-scale data deletion.
- Money Impact: Direct financial loss (e.g., erroneous payments, API credit exhaustion) exceeding the critical threshold.
- Reversibility: High friction or impossible; permanent data loss or irrevocable external API calls.
SEV-2: High / Partial Impairment
- Blast Radius: Significant subset of users affected; core agentic loop is broken for a primary use case.
- Data Exposure: Internal data leakage (cross-tenant) but restricted to a small number of users.
- Money Impact: Moderate financial burn or high risk of imminent financial loss.
- Reversibility: Difficult; requires manual database migrations or complex "undo" scripts to fix agent actions.
SEV-3: Medium / Degraded Performance
- Blast Radius: Intermittent failures; agent is producing "hallucinations" or low-quality output that slows down the user but doesn't break the system.
- Data Exposure: No external leak, but agent is ignoring privacy constraints in internal logs.
- Money Impact: Negligible direct loss, though operational costs may be slightly elevated due to retry loops.
- Reversibility: Easy; can be fixed by clearing a cache, updating a prompt, or restarting a session.
SEV-4: Low / Minor Issue
- Blast Radius: Edge case; affecting a single user or a non-critical feature.
- Data Exposure: None.
- Money Impact: None.
- Reversibility: Trivial.
Declaration and Escalation Paths
Ambiguity during the first 15 minutes of an incident is the primary driver of Mean Time to Resolve (MTTR). Clear ownership of the "Declare" button is essential.
Who declares?
The first responder (on-call engineer) has the authority and responsibility to declare the initial severity. If the responder is unsure, the default action must be to over-classify. It is safer to downgrade a SEV-1 to a SEV-2 after investigation than to treat a SEV-1 as a SEV-3 for an hour.
Escalation and Communication Cadence
- SEV-1: Immediate page to Engineering Leadership and Product Owners. Establish a dedicated war room. Communication updates every 30 minutes to stakeholders.
- SEV-2: Page the relevant feature team and the SRE lead. Communication updates every 2 hours.
- SEV-3: Ticket created and assigned to the team backlog. Updates provided via the internal status page or Slack channel daily.
- SEV-4: Logged as a bug. No immediate escalation required.
Navigating Borderline Calls
Agentic failures are often "grey." Use these worked examples to calibrate your team's judgment.
Example A: The "Looping" Agent
Scenario: An agent gets stuck in a logic loop, calling an external API 1,000 times per minute for a single user. It isn't crashing, but it's burning credits.
- The Borderline: Is this a SEV-2 (financial burn) or a SEV-3 (single user impact)?
- The Decision: If the API has a hard quota that could block all users once exhausted, this is a SEV-2. If the cost is negligible and isolated, it is a SEV-3.
Example B: The "Confident Hallucination"
Scenario: An agent is giving users incorrect technical advice. It isn't deleting data, but the advice is wrong 20% of the time.
- The Borderline: Is this a SEV-2 (core functionality broken) or a SEV-3 (degraded performance)?
- The Decision: Evaluate the "Danger of Action." If the advice could lead a user to run a destructive command on their own system, it is a SEV-2. If it's merely a typo or a slightly outdated fact, it is a SEV-3.
The Agent Triage Checklist
When an alert fires, the on-call engineer should run through this rapid-fire checklist before assigning a severity level:
Put it into practice
The Incident Triage Checklist Pack from IncidentLattice collects the templates and checklists behind this guide in ready-to-use form, so your team can apply the framework without drafting from scratch.