Small Changes, Big Production Failures

A production outage does not always start with a cyberattack or a dramatic infrastructure failure.
Sometimes it starts with something that looks completely routine: a deployment, a database change, a configuration update, or a change to authentication.

Recent incidents involving GitHub and Claude are a good reminder of this. However, the public incident records do not support treating all of these events as one shared failure or claiming a common root cause. GitHub reported a major incident on September 13, 2026, affecting about 28 services, while Claude's public status history shows separate incidents around the same period.
The more useful lesson is not "big companies keep making simple mistakes."

It is this:
In a complex system, a small change can expose a dependency or failure mode that was not obvious before.

---

What Happened With GitHub and Claude?​

GitHub's September 13 incident affected approximately 28 services, including Issues, Pull Requests, Actions, Codespaces, Pages, Notifications, Code Scanning, Git LFS, and new account signup.

At the peak of the incident, 8.8% of requests to create GitHub App installation access tokens failed, while token issuance for Actions workflows affected about 4% of workflows during the incident. GitHub said it would publish a detailed root-cause analysis.

GitHub also experienced a separate incident on September 4 involving Copilot code review. That incident was caused by a change to authentication permissions that prevented the service from submitting affected reviews to the GitHub API. GitHub reverted the change and restored normal operation.

Claude's status history shows several separate incidents in September, including elevated errors affecting specific models on September 15 and September 11, as well as other service issues. Its September 16 incident concerned Google Play subscriptions rather than a general Claude authentication outage.

So there is an important distinction:
These incidents should not be presented as one confirmed technical failure or as proof of a shared root cause.
But they do illustrate the same broader engineering problem: complex systems contain dependencies that can make apparently small changes surprisingly expensive.

---

Why Can a Small Change Break a Large System?​

Large production systems rarely consist of isolated components.
A seemingly simple change can interact with:

• Database schemas
• Authentication services
• APIs
• Configuration
• Permissions
• Deployment pipelines
• Caches
• Queues
• External services
• Monitoring and alerting
• Other internal services

For example, imagine that an application depends on a database index.
The index itself might look like a small implementation detail. But if application queries, migrations, monitoring, or another service implicitly depend on it, removing that index can have consequences far beyond the database.
The same principle applies to authentication.
A small permissions change can prevent one service from obtaining the credentials or access tokens it needs. GitHub's September 4 Copilot incident is a real example of this type of dependency.

---

Silent Failures Can Be Worse Than Visible Errors​

One of the most dangerous situations in production is not always a visible error.
Sometimes the system returns a successful response even though the expected operation did not actually complete correctly.
That can make the problem harder to detect because:

• Monitoring may report the request as successful.
• Users may not immediately see an error.
• Logs may not contain an obvious failure.
• The problem can spread before anyone notices it.

This is why production monitoring should not only ask:
"Did the request return an error?"

It should also ask:
"Did the expected business operation actually succeed?"

Those are two different questions.

---

Cascading Failures Make Small Problems Bigger​

A production system can also fail through a chain reaction.
For example:

1. One service becomes unhealthy.
2. Other services retry requests.
3. Retry traffic increases the load.
4. Another dependency becomes overloaded.
5. More requests begin failing.
6. The failure spreads to additional services.

This is one reason distributed systems need clear failure boundaries.
A problem in one component should not automatically become a problem for every component that depends on it.

Useful techniques include:

• Timeouts
• Rate limits
• Circuit breakers
• Retry limits
• Backoff strategies
• Queue-based processing
• Health checks
• Isolation between critical services
• Graceful degradation

The exact solution depends on the architecture, but the principle is simple:
Design the system so that one failure has a limited blast radius.

---

What Can Developers Learn From These Incidents?​

You do not need millions of users for these lessons to matter.
Even a small production application can benefit from the same engineering principles.

Treat Every Production Change as a Risk​

A deployment may be routine, but it can still expose an existing weakness.
The same applies to:

• Database migrations
• Configuration changes
• Permission changes
• Dependency upgrades
• Infrastructure changes
• Authentication changes

Before making a change, ask:
What else depends on this?

Keep Schema and Application Code in Sync​

If application code expects a database object to exist, that dependency needs to be understood and managed.

Do not assume that removing an apparently unused index, column, table, permission, or configuration value is harmless.

Before changing the schema, check:

• Application queries
• Background jobs
• Reports
• Monitoring
• Scheduled tasks
• Other services
• Deployment and migration logic

Test More Than the HTTP Response​

A successful HTTP response does not necessarily mean that the intended operation succeeded.
For important workflows, monitor meaningful outcomes.
For example, if a service creates an access token, the monitoring system should help detect failures in token creation—not simply confirm that an endpoint responded with HTTP 200.

Design for Failure​

Assume that a dependency will eventually become unavailable.
Ask:

• What happens if the database is unavailable?
• What happens if the network becomes unreliable?
• What happens if a third-party API stops responding?
• What happens if traffic suddenly increases?
• What happens if a deployment introduces a regression?
• What happens if one service becomes unhealthy?

These questions are part of resilience engineering, not just disaster recovery.

---

Even Large Engineering Teams Have Production Incidents​

GitHub and GitLab provide useful historical examples of why operational resilience matters.
GitLab's January 2017 database outage was caused by accidental removal of data from its primary database server. GitLab's postmortem explains that replication had stopped and the recovery process involved manual work that was not sufficiently automated or documented. The outage lasted many hours and resulted in permanent loss of some production data.

The important lesson is not that experienced engineers are careless.
It is that complex production systems can fail in ways that are difficult to anticipate.

That is why mature engineering practices focus heavily on:

• Failure recovery
• Automation
• Observability
• Testing
• Backups
• Deployment safety
• Dependency management
• Incident response
• Failure isolation

---

Build Your System Assuming It Can Fail​

The goal of good engineering is not to create a system that can never fail.
That is unrealistic.
The goal is to create a system where failures are:

• Detected quickly
• Contained when possible
• Recoverable
• Understandable
• Less likely to become catastrophic

A useful mindset is:
Expect failure. Limit the blast radius. Detect it early. Recover quickly.
The difference between a fragile system and a resilient one is not that the resilient system never breaks.
It is that the team has already considered what happens when something does break.

---

Frequently Asked Questions​

-----------------

Can a small code change cause a major production outage?​

Yes. A small change can affect another component through an undocumented dependency, configuration, permission, schema, or operational assumption. The size of the change does not necessarily reflect the size of its impact.

Are production outages usually caused by hackers?​

Not necessarily. Production incidents can result from software changes, configuration problems, infrastructure failures, dependency failures, data issues, or security incidents. Each outage needs to be analyzed from its documented evidence rather than assuming a particular cause.

What is a cascading failure?​

A cascading failure occurs when a failure in one component causes additional failures in dependent components, allowing the original problem to spread through the system.

How can developers reduce the impact of production failures?​

Use appropriate techniques such as monitoring, health checks, timeouts, rate limits, circuit breakers, controlled deployments, backups, automated recovery, and clear service boundaries. The right combination depends on the system's architecture and failure modes.