Production Outage: What to Do First

If production goes down right now, what's the first thing you should do?
Most developers will say: "Check the logs and find the root cause."
Sounds reasonable, right? But Google's Site Reliability Engineering (SRE) guidance challenges that instinct.
Your first priority isn't finding the bug. It's restoring the service.
Why? Because while you're investigating the root cause, your customers are still experiencing an outage. Finding the problem doesn't help them if the system is still down.

---

Here's the right approach. 🚨

1. Restore the Service First​

Focus on getting production back online, even if you don't know exactly what caused the outage.
Depending on the situation:

• 🔄 Recent deployment? Roll back to the last known good version.
• 🚩 New feature causing issues? Disable it using a feature flag.
• 🖥️ One cluster is down? Redirect traffic to healthy clusters.
• 📈 System overloaded? Disable nonessential workloads to reduce pressure.

The goal is to stop the impact on customers as quickly as possible.

---

2. Preserve Evidence and Communicate​

Before making changes that could destroy useful evidence, capture relevant logs, metrics, and system state.
You'll need this information later to understand what happened.
Also, keep your team informed:

• Explain what you know.
• Share what you're doing.
• Assign clear responsibilities.
• Avoid having multiple engineers make conflicting changes.

Good communication prevents an outage from becoming even worse. 😅

---

3. Find the Root Cause​

Once the service is stable again, start investigating.
A good starting point is the most recent change:

• Code deployments.
• Configuration changes.
• Infrastructure updates.
• Traffic or workload changes.

Look for what changed shortly before the outage. Often, a previously healthy system starts failing because something around it changed.

---

4. Write a Blameless Postmortem​

After resolving the incident, document what happened.
A good postmortem should explain:

• What happened?
• What caused the incident?
• How was it detected?
• How was service restored?
• What can prevent it from happening again?

Focus on improving the system, not blaming individuals.
The goal is to understand what allowed the failure to happen and prevent similar incidents in the future.

---

💡 What's the Interviewer Really Looking For?​

This question isn't just testing your debugging skills.
The interviewer wants to know whether you understand production responsibilities.
A strong answer demonstrates that you:

• Prioritize customer impact over debugging.
• Know how to mitigate an outage quickly.
• Understand incident communication.
• Can investigate root causes after recovery.
• Believe in blameless postmortems.

These skills matter whether you're interviewing for a DevOps, Backend, SRE, or QA Engineer position.
Remember: In production, restoring the service comes before finding someone or something to blame.