lesson
55 min

Incident Response

Plan for AI failures before they happen: severity levels, a kill switch, communication, and a blameless review that leads to real fixes.

Objective

By the end of this lesson, you can write an incident response runbook for an AI feature, including severity levels, a kill switch, who communicates with whom, and how to run a blameless review.

AI incidents are different

Traditional incidents are often loud: the site is down. AI incidents are often quiet: the assistant gives wrong eligibility answers in Spanish for three days, and nobody notices until a family is turned away. That is why monitoring (last lesson) and a clear way for staff and clients to report problems matter so much.

Examples of AI incidents:

  • Wrong answers that could lead someone to miss a deadline or a benefit.
  • Personal data shown to the wrong person.
  • Harmful or offensive output.
  • An agent taking an action nobody approved.
  • A successful prompt injection.

Severity levels

Decide levels in advance so nobody argues during the incident:

LevelMeaningExampleResponse
SEV 1Harm to people or data exposureClient records shown to another userKill switch now; leadership and legal notified
SEV 2Wrong answers on important topicsWrong ID rules in one languageDisable affected topic or language; fix within a day
SEV 3Degraded but safeSlow responses, fallbacks workingFix in normal work

The kill switch

A kill switch turns off the AI feature quickly and sends people to a safe path. Requirements:

  • One action, by a named role, with no code deployment needed (a feature flag).
  • The fallback is ready: FAQ page, fixed replies, phone line.
  • It is tested on a schedule. An untested kill switch is a hope.
  • Partial switches are useful too: turn off one tool, one topic, or one language.

The runbook

A runbook is the step-by-step guide for the person on call:

  1. Detect and declare. Anyone can raise an incident. Name an incident lead.
  2. Contain. Use the kill switch or partial switch. Stop the harm first, investigate second.
  3. Assess. Who was affected? What data? Over what time? Logs answer this (last lesson).
  4. Communicate. Tell staff what to say. Notify affected people honestly and in their language. Follow legal notification rules for data exposure.
  5. Fix and verify. Add the failure to the test set. Confirm the fix passes before turning the feature back on.
  6. Review. Hold a blameless review within a week.

Blameless reviews

A blameless review asks how the system allowed the failure, not who to punish. People report problems honestly when reporting is safe. The output is a short write-up: timeline, impact, root causes, and action items with owners and dates.

A national example

Incident readiness

0 of 8 done

Tabletop drill

Private. Saved only in this browser. No one else can see it.

Go deeper: Systems Administration, Simplified: A Practical Blueprint for Uptime, Security, and Scale

Ask. Answer. Build together.