Major Incident Management Drill =============================== Slide 1: Major Incident Management Drill Narration Anna: Welcome to our major incident drill. Think of it as a fire drill for IT—everyone needs to know their part before the real emergency hits. Greg: We'll walk through a simulated outage so you can practice the motions without the adrenaline spike. The goal is to stay calm and restore service swiftly. On-screen text Major Incident Management Drill Staying calm when systems fail Slide 2: When is it a major incident? Narration Greg: A major incident is more than a glitch. It affects core services and usually drags multiple teams into the fight. Picture the checkout system rejecting every payment during a sale—that's major. Anna: When you see widespread impact or a breached SLA on the horizon, escalate it and rally the right experts immediately. On-screen text When is it a major incident? - Impacts critical business services - Requires cross-team coordination - Needs immediate communication Slide 3: Example scenario Narration Anna: Example scenario focuses attention on a concrete part of the work. Checkout system fails across all regions, Users can't complete purchases, and Mobilize database, network and app teams. Greg: In practice, ask who owns the work, what evidence proves it happened, and what handoff comes next. Use the supporting details as a checklist: Users can't complete purchases; Mobilize database, network and app teams. On-screen text Example scenario - Checkout system fails across all regions - Users can't complete purchases - Mobilize database, network and app teams Slide 4: Checkout architecture Narration Anna: Let's map the checkout flow so everyone knows where problems can start. The web front end hits an API gateway, which then fans out to microservices for payments, inventory, and orders. Those services talk to a clustered database and ping the payment provider over the internet. Monitoring agents watch each step. On-screen text Checkout architecture - Web front end served from CDN - API gateway routing to microservices - Payment gateway integration - Inventory and order databases - Monitoring at each layer Slide 5: Who is involved? Narration Greg: A failure anywhere in that chain kicks off a big response. The incident commander coordinates efforts and sets priorities. A communications lead keeps executives and customers in the loop. Front-end and backend engineers dig into code issues. Database admins check queries and replication. Network ops verify connectivity and DNS. A liaison talks to the payment provider. Finally, the service desk fields user reports and keeps them updated. On-screen text Who is involved? - Incident commander coordinates the response - Communications lead keeps stakeholders informed - Front-end and backend engineering teams - Database administrators - Network operations - Payment provider liaison - Service desk and customer support Slide 6: Running the drill Narration Anna: During the drill, assign each role quickly and follow the runbook. The incident commander leads, comms handles updates, and a scribe logs every step. Greg: Treat it as the real thing—no shortcuts. The more accurately you rehearse, the smoother an actual outage will play out. On-screen text Running the drill - Assign clear roles fast - Follow the runbook step by step - Capture decisions and timelines Slide 7: After-action review Narration Greg: Once the dust settles, gather the team to review what happened. Share successes, missteps, and any surprises. Anna: Update the playbook based on those lessons and book another drill. Practice turns panic into muscle memory. On-screen text After-action review - Share what went well and what didn't - Update the playbook accordingly - Schedule the next practice