Anna: Welcome to our major incident drill. Think of it as a fire drill for IT—everyone needs to know their part before the real emergency hits.
Greg: We'll walk through a simulated outage so you can practice the motions without the adrenaline spike. The goal is to stay calm and restore service swiftly.
Greg: A major incident is more than a glitch. It affects core services and usually drags multiple teams into the fight. Picture the checkout system rejecting every payment during a sale—that's major.
Anna: When you see widespread impact or a breached SLA on the horizon, escalate it and rally the right experts immediately.
Anna: Example scenario focuses attention on a concrete part of the work. Checkout system fails across all regions, Users can't complete purchases, and Mobilize database, network and app teams.
Greg: In practice, ask who owns the work, what evidence proves it happened, and what handoff comes next. Use the supporting details as a checklist: Users can't complete purchases; Mobilize database, network and app teams.
Anna: Let's map the checkout flow so everyone knows where problems can start.
The web front end hits an API gateway, which then fans out to microservices for
payments, inventory, and orders. Those services talk to a clustered database and
ping the payment provider over the internet. Monitoring agents watch each step.
Greg: A failure anywhere in that chain kicks off a big response.
The incident commander coordinates efforts and sets priorities.
A communications lead keeps executives and customers in the loop.
Front-end and backend engineers dig into code issues.
Database admins check queries and replication.
Network ops verify connectivity and DNS.
A liaison talks to the payment provider.
Finally, the service desk fields user reports and keeps them updated.
Anna: During the drill, assign each role quickly and follow the runbook.
The incident commander leads, comms handles updates, and a scribe logs every step.
Greg: Treat it as the real thing—no shortcuts. The more accurately you rehearse,
the smoother an actual outage will play out.
Greg: Once the dust settles, gather the team to review what happened. Share successes, missteps, and any surprises.
Anna: Update the playbook based on those lessons and book another drill. Practice turns panic into muscle memory.