Your first production incident: the 30-minute playbook

By Carl Mills - senior engineer, veteran of many 3am pages • Published 9 July 2026 • Last updated 9 July 2026

Minute 0-5: confirm and communicate

Two things, in parallel, before any debugging:

1. Confirm the blast radius. Is it down for everyone or just you? Check from your phone off wifi, or a checker site. Is it the whole site, one page, one feature? "Checkout is erroring for all users" and "one customer can't log in" are different emergencies.
2. Say something now. The message that marks you as trustworthy under pressure is boring: "Site is returning 500s since 14:20. I'm investigating. Next update by 15:00." No cause speculation, no promises. Silence is the only unforgivable move - people assume nobody is on it.

Minute 5-15: find "what changed?"

Production systems that worked yesterday rarely break spontaneously. In rough order of likelihood:

Then read the logs from the moment it broke - not the code. The error that appears 400 times a minute since 14:20 is your thread to pull.

Minute 15-30: roll back by default

The junior instinct is to fix the bug live, under pressure, in production. The senior move is usually to make the impact stop first:

• If a deploy broke it: roll back to the previous release. Every platform has this - Vercel/Netlify "redeploy previous", Azure "swap slots" or redeploy, or git revert + deploy.
• If a config change broke it: change it back, even if you don't yet understand why.
• If a provider broke it: you're in communication mode, not fix mode - update users and watch the status page.

Rolling back isn't admitting defeat; it converts a live crisis into an offline puzzle. Users get their service back in minutes, and you get unlimited calm time to find the root cause.

The never-do list

• Never debug in silence. An update every 30 minutes, even "still looking", keeps trust intact.
• Never hot-edit production without a checkpoint. Note what you're changing before you change it - panic edits cause second incidents.
• Never delete anything during an incident. Not logs, not data, not the "obviously broken" thing. You'll want it for diagnosis.
• Never guess in your comms. "Probably the database" said at minute 10 becomes "they don't know what they're doing" at minute 60 when it wasn't.

Afterwards: the postmortem that makes you look senior

Within a day or two, write one page, blamelessly: what happened (timeline), impact (who, what, how long), root cause (the system gap, not the person), and 2-3 prevention actions with owners. The framing that shows maturity: if one person's mistake can take production down, the missing safety net is the real bug. Teams remember who wrote the calm, honest postmortem far longer than who caused the outage.

Frequently asked questions

What if I caused it?

Say so early and plainly - "my deploy at 14:18 looks like the trigger, rolling back now" - and you'll gain more credibility than you lose. Cover-ups, not mistakes, end careers.

When do I escalate and wake someone up?

When money or data is actively at risk, when you're past your own access or knowledge, or when 30 minutes of triage has produced nothing. Escalating at the right time reads as judgment, not weakness.

We're a tiny team with no monitoring - where do we start?

One free uptime check (many services ping your site every minute and email on failure), error alerts from your platform, and a written 5-line "who does what" plan. An hour of setup halves the length of your next incident - it's also step 12 of our launch checklist.