The 5 Whys technique is one of the most powerful tools in root cause analysis, and one of the most commonly misused. Developed at Toyota as part of the Toyota Production System, it's deceptively simple: keep asking "why" until you reach the underlying cause. But simplicity doesn't mean easy. Here's how to use 5 Whys effectively in DevOps contexts.
What is the 5 Whys Technique?
The 5 Whys is an iterative interrogation technique. You start with a problem statement and ask "why did this happen?" For each answer, you ask "why?" again until you reach a root cause that you can actually address.
The number five is a guideline, not a rule. Some problems are solved in three whys. Others need seven or eight. The goal is to go deep enough to find causes you can fix, not to hit an arbitrary count.
Key Principle
A good 5 Whys analysis should end at a cause you can control. If your analysis ends at "because users are stupid" or "because the vendor made a mistake," you haven't reached an actionable root cause. Keep asking: why did users make that choice? Why didn't we protect against the vendor issue?
5 Whys Template for DevOps Teams
Use this template to structure your 5 Whys analysis. Fill in each level, and don't move to the next "why" until you've verified the current answer is accurate.
5 Whys Analysis Template
Incident
[Describe the incident in one sentence]
Impact
[Duration, affected users/services, business impact]
Why #1
[What was the immediate technical cause?]
Why #2
[Why did that happen?]
Why #3
[Why did that happen?]
Why #4
[Why did that happen?]
Why #5
[Why did that happen?]
Root Cause
[Summarize the root cause(s) identified]
Corrective Actions
[List specific actions with owners and due dates]
Real-World DevOps Examples
Example 1: Production Database Outage
Incident: Production database became unresponsive for 45 minutes
Why #1: Why was the database unresponsive?
The database ran out of available connections.
Why #2: Why did it run out of connections?
A new API endpoint was creating connections but not releasing them.
Why #3: Why wasn't it releasing connections?
The developer used a manual connection instead of the connection pool, and didn't close it in the error path.
Why #4: Why wasn't this caught in code review?
The reviewer wasn't familiar with our database patterns, and we don't have automated checks for connection handling.
Why #5: Why don't we have automated checks?
We haven't established standard database patterns or linting rules for connection management.
Root Causes:
- No enforced standards for database connection handling
- No automated detection of connection leaks
- Knowledge gap in code review process
Actions:
- Create and document database connection standards
- Add lint rule to detect manual connection usage
- Add connection pool monitoring and alerting
- Include database patterns in onboarding
Example 2: Deployment Caused Service Degradation
Incident: API latency increased 10x after deployment
Why #1: Why did latency increase after deployment?
The new version was making N+1 database queries.
Why #2: Why was it making N+1 queries?
A refactored function moved a batch query into a loop.
Why #3: Why wasn't this caught before production?
Our staging environment has much less data than production, so the query count wasn't noticeable.
Why #4: Why is staging data so different from production?
We don't have a process for keeping staging data realistic.
Why #5: Why don't we have canary deployments to catch this?
Our deployment pipeline doesn't support gradual rollouts.
Root Causes:
- Staging environment not representative of production scale
- No gradual rollout capability to catch performance regressions
- No query count monitoring in deployment validation
Example 3: Alert Fatigue Led to Missed Incident
Incident: Customers reported outage 30 minutes before team noticed
Why #1: Why didn't the team notice for 30 minutes?
The critical alert was buried in a flood of other alerts.
Why #2: Why were there so many other alerts?
We have 200+ alerts, and many fire frequently without requiring action.
Why #3: Why do alerts fire without requiring action?
Alert thresholds were set too aggressively, and nobody has tuned them.
Why #4: Why hasn't anyone tuned the alerts?
There's no owner for alert hygiene, and no process for reviewing alert effectiveness.
Root Causes:
- No alert ownership or hygiene process
- Alerts not prioritized by severity
- No regular review of alert signal-to-noise ratio
Common 5 Whys Mistakes
1. Stopping Too Early
The most common mistake. If your 5 Whys ends at "the config was wrong," you haven't found the root cause. Why was it wrong? Why wasn't it caught? What systemic issue allowed this?
2. Following Only One Path
Most incidents have multiple contributing factors. If you only follow one "why" chain, you miss the others. Consider branching your analysis when there are multiple valid answers to "why."
Tip: Multi-Path Analysis
When a single "why" has multiple valid answers, explore each path. This often reveals that what seemed like a single incident was actually the intersection of multiple systemic issues. Tools that support branching cause chains make this easier to manage and document.
3. Blaming People
"Because the developer made a mistake" is not a root cause. People always make mistakes. The question is: what allowed the mistake to reach production? What guardrails were missing?
4. Accepting Unverified Answers
Each "why" answer should be verified with evidence. Logs, metrics, code commits, chat transcripts. If you're guessing at answers, your root cause analysis is built on assumptions.
5. Ending at External Factors
"The third-party service went down" is not something you can fix. But you can fix: why didn't we have redundancy? Why wasn't there a fallback? Why didn't we detect it faster?
Tips for Better 5 Whys
- Build the timeline first. You can't analyze causes accurately without understanding what happened and when. Start with a detailed chronology.
- Involve the right people. Include those who were on-call, those who wrote the code, and those who understand the architecture. Different perspectives reveal different causes.
- Write it down as you go. Don't rely on memory. Document each "why" and its answer during the discussion, not after.
- Look for systemic issues. Individual mistakes are symptoms. Look for the process, tooling, or cultural factors that allowed them.
- End with actions. Every 5 Whys should produce specific, assigned corrective actions. Otherwise, you've understood the problem but done nothing to fix it.
Combining 5 Whys with Other Techniques
5 Whys works well alone, but it's even more powerful combined with other RCA methods:
- Timeline analysis helps ensure you're asking "why" about the right things
- Fishbone diagrams ensure you're considering causes across multiple categories
- Problem definition (4H1W) ensures you've clearly scoped what you're analyzing
The best RCA processes use multiple tools depending on the incident complexity. Simple incidents might need only 5 Whys. Complex incidents benefit from a full toolkit.
Key Takeaways
- Keep asking "why" until you reach causes you can control and fix
- Verify each answer with evidence before moving to the next "why"
- Focus on systems and processes, not individual blame
- Explore multiple paths when incidents have multiple contributing factors