Anna: Welcome to our major incident drill. Think of it as a fire drill for IT—everyone needs to know their part before the real emergency hits.
Greg: We'll walk through a simulated outage so you can practice the motions without the adrenaline spike. The goal is to stay calm and restore service swiftly.
Greg: A major incident is more than a glitch. It affects core services and usually drags multiple teams into the fight. Picture the checkout system rejecting every payment during a sale—that's major.
Anna: When you see widespread impact or a breached SLA on the horizon, escalate it and rally the right experts immediately.
Anna: Here's the scenario for the drill. The checkout system fails across all regions, simultaneously.
Greg: Which means users can't complete purchases anywhere. Revenue stops, and it stops visibly, so the pressure arrives within minutes.
Anna: It also crosses team boundaries immediately. Database, network and application teams all need to be mobilised, and none of them can diagnose it alone.
Greg: That's exactly why it qualifies as a major incident. It's not the severity of any one component failing, it's that the response requires coordination nobody has time to invent on the spot.
Anna: Let's map the checkout flow so everyone knows where problems can start.
The web front end hits an API gateway, which then fans out to microservices for
payments, inventory, and orders. Those services talk to a clustered database and
ping the payment provider over the internet. Monitoring agents watch each step.
Greg: A failure anywhere in that chain kicks off a big response.
The incident commander coordinates efforts and sets priorities.
A communications lead keeps executives and customers in the loop.
Front-end and backend engineers dig into code issues.
Database admins check queries and replication.
Network ops verify connectivity and DNS.
A liaison talks to the payment provider.
Finally, the service desk fields user reports and keeps them updated.
Anna: A P1 response needs a cadence before it needs a perfect diagnosis. Declare the incident, open one record and assign the core roles quickly.
Greg: Then agree when the next update is due and follow the current runbook. The commander can change the cadence when impact or evidence changes, but the team should not improvise five separate response plans.
Anna: Test the safest recovery path, verify that service has actually returned, and hand off every remaining action with an owner.
Greg: Rehearse those steps without shortcuts. The incident record should show what happened, what was decided and what still needs work.
Anna: Four roles keep the response coordinated. The incident commander sets priorities, approves risk and decides when the team will report again.
Greg: Communications turns confirmed facts into updates for staff, customers and executives. That person always states the known impact and the next update time.
Anna: The resolver coordinates specialists, tests recovery options and verifies that service is back. The scribe records timestamps, evidence, decisions, owners and open questions.
Greg: Those people work from one incident record. Specialists may rotate as the diagnosis changes, but the four responsibilities stay visible and the record becomes evidence for the review.
Greg: Once the dust settles, gather the team to review what happened. Share successes, missteps, and any surprises.
Anna: Update the playbook based on those lessons and book another drill. Practice turns panic into muscle memory.