Chapter 12: Incident Response and Troubleshooting
Learning Objectives
After completing this chapter, you will be able to:
- Respond effectively to infrastructure incidents using structured methodologies
- Apply systematic troubleshooting approaches to complex technical problems
- Conduct thorough root cause analysis that drives meaningful improvements
- Implement comprehensive incident management processes aligned with ITIL practices
- Develop and maintain actionable runbooks that enable rapid response
- Lead incident response teams effectively through critical situations
Introduction
Infrastructure incidents are inevitable occurrences that impact business operations, user experience, and organizational reputation. The modern digital enterprise operates in an environment where even brief service disruptions can cascade into significant financial losses, customer dissatisfaction, and competitive disadvantage. The difference between a minor disruption that is quickly contained and a major outage that damages business outcomes often comes down to a single critical factor: how well prepared teams are to respond when things go wrong.
Effective incident response represents far more than simply reacting to problems as they arise. It encompasses a comprehensive discipline that combines technical expertise with organizational processes, communication protocols, and continuous learning mechanisms. When infrastructure issues occur—whether they manifest as application errors, performance degradation, security breaches, or complete service failures—the organization’s ability to detect, diagnose, and resolve these incidents quickly and effectively becomes a core competency that directly impacts business success.
This chapter explores the full spectrum of incident management capabilities that infrastructure and platform teams must develop and maintain. We examine not only the tactical skills required to troubleshoot specific technical problems, but also the strategic frameworks that enable organizations to build resilient operations. From the initial moments when an incident is detected through the final stages of learning and improvement, we provide practical guidance that teams can apply immediately while also building long-term organizational capability.
The practices described here draw from proven methodologies including ITIL incident management, DevOps principles, and Site Reliability Engineering (SRE) approaches. We integrate these frameworks into practical guidance tailored specifically for infrastructure and platform management contexts. Throughout this exploration, we emphasize both the human and technical dimensions of incident response, recognizing that effective resolution requires not just the right tools and processes, but also the right team dynamics, communication patterns, and organizational culture.
Incident Management Process
Understanding the Incident Lifecycle
Every infrastructure incident, regardless of its specific nature or severity, follows a predictable lifecycle from initial detection through final resolution and learning. Understanding this lifecycle enables teams to structure their response activities, assign appropriate resources at each stage, and ensure that no critical steps are overlooked during the pressure of an active incident. The incident lifecycle represents a continuous flow rather than a series of discrete, disconnected activities, with each phase naturally leading into the next while maintaining opportunities for feedback loops and iteration.
The lifecycle begins with detection, the critical first moment when the organization becomes aware that something is wrong. Detection may occur through multiple channels: automated monitoring systems that identify anomalies in metrics or logs, user reports submitted through support channels, or proactive discovery by team members during routine operations. The speed and accuracy of detection directly correlates with overall incident impact—early detection of emerging problems enables intervention before issues escalate to full service disruption.
Following detection, incidents enter the triage phase where responders make critical initial assessments that shape the entire response effort. During triage, teams evaluate the nature and scope of the problem, classify the incident according to severity and impact, determine the appropriate priority level, and identify which teams and resources should be mobilized. Effective triage requires balancing the need for rapid action against the necessity of gathering sufficient information to make informed decisions. Teams that rush through triage risk mobilizing inappropriate resources or applying wrong solutions, while teams that over-analyze during this phase allow incidents to worsen while deliberation continues.
The response phase represents the heart of incident management activities. During this extended period, technical teams actively work to diagnose root causes, develop and test potential solutions, implement fixes or workarounds, and manage the complex coordination required when multiple teams must work together. Response activities follow systematic troubleshooting methodologies rather than random trial-and-error approaches, with teams forming hypotheses based on available evidence, testing those hypotheses through structured investigation, and iteratively narrowing the problem space until root causes are identified and addressed.
Recovery extends beyond simply implementing a fix to encompass the full restoration of normal service operations. During recovery, teams verify that implemented solutions have actually resolved the incident, confirm that services meet their defined performance and availability targets, remove any temporary workarounds that were implemented during active response, and transition the environment back to steady-state operations. Recovery also includes the critical validation that user-facing functionality has been restored and that business operations can resume normally.
+-----------------------------------------------------------------------+
| INCIDENT LIFECYCLE |
+-----------------------------------------------------------------------+
| |
| +----------+ +----------+ +----------+ +----------+ |
| | DETECT |---->| TRIAGE |---->| RESPOND |---->| RECOVER | |
| | | | | | | | | |
| | - Alerts | | - Assess | | - Diagnose | - Restore | |
| | - Users | | - Classify | - Mitigate | - Verify | |
| | - Monitor| | - Prioritize | - Communicate | - Document| |
| +----------+ +----------+ +----------+ +----------+ |
| | | |
| | v |
| | +----------+ |
| +<-------------------------------------------| LEARN | |
| | | |
| Continuous Improvement | - Review | |
| | - Improve| |
| | - Prevent| |
| +----------+ |
| |
+-----------------------------------------------------------------------+
The final phase, learning, completes the cycle by extracting value from the incident experience. This phase involves conducting blameless postmortems that examine what occurred without assigning personal blame, identifying systemic improvements that could prevent similar incidents or improve response effectiveness, implementing concrete action items derived from lessons learned, and sharing knowledge across the organization. Learning transforms incidents from purely negative events into opportunities for organizational growth and capability development.
Incident Classification and Prioritization
Effective incident management requires a systematic approach to classification and prioritization that enables teams to allocate resources appropriately and set realistic expectations with stakeholders about response and resolution timelines. Classification systems must balance precision with practicality—overly complex classification schemes create confusion and inconsistency, while oversimplified approaches fail to capture important distinctions that should influence response approaches.
Priority classification typically considers two fundamental dimensions: impact (how severely the incident affects business operations and users) and urgency (how quickly the situation may worsen without intervention). The combination of these factors produces a priority level that dictates response expectations. Critical priority incidents represent situations where core business operations have stopped or where severe data loss is occurring. These situations demand immediate response, typically within fifteen minutes of detection, with resources marshaled to achieve resolution within one to four hours depending on the specific situation.
High priority incidents involve major service degradation that significantly impairs business operations without causing complete failure. These incidents might manifest as partial outages affecting important but not critical services, severe performance degradation that makes systems barely usable, or situations affecting large numbers of users even if alternative workarounds exist. High priority incidents typically require response within thirty minutes and target resolution within four to eight hours.
Medium priority incidents represent situations with limited business impact where core services remain operational but specific features or non-critical capabilities are impaired. These might include issues affecting small user populations, problems for which effective workarounds exist, or degradation that is noticeable but doesn’t significantly impair business operations. Medium priority incidents generally allow response times of two to four hours with resolution targets of twenty-four to forty-eight hours.
Low priority incidents encompass minimal-impact situations such as cosmetic issues, problems affecting only test environments, or situations where users experience minor inconveniences that don’t materially affect their ability to accomplish business objectives. These incidents typically allow response times measured in business days rather than hours, with resolution expectations of several days to a week.
| Priority | Impact | Urgency | Response Time | Resolution Target | Examples |
|---|---|---|---|---|---|
| P1 Critical | Business stopped | Immediate | < 15 min | 1-4 hours | Production completely down, active data loss, security breach |
| P2 High | Major degradation | High | < 30 min | 4-8 hours | Partial outage, severe performance issues affecting many users |
| P3 Medium | Limited impact | Medium | < 2 hours | 24-48 hours | Non-critical service affected, workarounds available |
| P4 Low | Minimal impact | Low | < 8 hours | 3-5 days | Cosmetic issues, minor bugs with easy workarounds |
The incident severity matrix provides a practical framework for making priority determinations based on the scope of user impact and the degree of business consequence. Critical incidents combine wide user impact with high business consequences—situations where many users are affected and the business impact is severe. High priority incidents involve either wide impact with lower business consequences or high business consequences affecting a narrow user base. Medium priority incidents typically affect many users with minimal business impact, while low priority incidents combine narrow impact with low business consequences.
This classification framework must be adapted to each organization’s specific context. A commercial e-commerce platform might classify shopping cart functionality failures as critical incidents due to direct revenue impact, while the same organization might consider employee productivity tool issues as medium priority. Healthcare organizations might elevate incidents affecting patient care systems to critical priority regardless of the number of users affected, reflecting the potentially life-threatening consequences of healthcare system failures.
Incident Response Roles and Responsibilities
Effective incident response requires clear role definition and assignment, ensuring that all necessary activities receive appropriate attention without creating confusion, duplication of effort, or gaps in coverage. The incident command system provides a proven organizational structure that scales from small incidents handled by a single responder to major incidents requiring coordination across multiple teams and even multiple organizations.
The incident commander serves as the central coordination point for the entire response effort. This role requires strong leadership and decision-making capabilities rather than necessarily requiring the deepest technical expertise. The incident commander maintains overall situational awareness, makes key decisions about response strategy and resource allocation, manages communication to executive leadership and external stakeholders, and determines when to escalate or when to declare an incident resolved. Effective incident commanders balance decisive action with appropriate delegation, recognizing that they cannot and should not attempt to perform all response activities personally.
The technical lead focuses specifically on the diagnostic and resolution activities required to address the incident’s technical root causes. While the incident commander manages the overall response, the technical lead directs the investigation, formulates and tests hypotheses about potential causes, coordinates the work of multiple technical resources when necessary, and ultimately implements or oversees implementation of the solution. The technical lead should possess deep expertise in the affected systems and technologies, enabling them to quickly narrow the problem space and identify promising investigation paths.
The communications lead manages all stakeholder communication during the incident, freeing the incident commander and technical resources to focus on resolution activities. This role involves crafting initial incident notifications, providing regular status updates at appropriate intervals, maintaining the public status page if one exists, coordinating with customer support teams who may be fielding user inquiries, and preparing final resolution communications. Effective communications leads understand how to translate technical details into business language appropriate for various audiences while maintaining transparency about what is known, what remains uncertain, and what actions are being taken.
The scribe documents the incident timeline, capturing key events, decisions, and actions taken throughout the response effort. This documentation serves multiple critical purposes: it provides material for the postmortem analysis, creates an audit trail for compliance purposes, enables team members joining the incident later to quickly understand what has already occurred, and preserves institutional memory about how similar situations have been handled. The scribe role, while sometimes undervalued during the high-pressure incident response, often proves invaluable during postmortem activities when teams attempt to reconstruct exactly what happened and when.
Subject matter experts provide specialized knowledge about specific technologies, systems, or business domains relevant to the incident. These individuals typically join the response effort when initial investigation identifies that their expertise is needed, contribute their specialized knowledge to diagnosis and resolution, and then disengage once their contribution is complete. Organizations should maintain clear rosters of subject matter experts across different technology domains, ensuring that responders can quickly identify and engage the right expertise when needed.
On-Call Practices and Escalation
On-call rotations represent the human infrastructure that enables around-the-clock incident response capability. Effective on-call practices balance organizational needs for continuous coverage against individual needs for sustainable work-life balance. Organizations that fail to structure on-call responsibilities appropriately face burnout, decreased response effectiveness, and ultimately inability to retain the skilled staff required for effective operations.
The primary on-call engineer serves as the first responder for all alerts and incidents occurring during their shift. This individual receives initial notifications, performs initial triage and assessment, handles straightforward incidents independently when possible, and escalates to secondary on-call or specialized resources when necessary. Primary on-call responsibilities typically rotate among team members in weekly shifts, with careful attention to providing adequate rest periods between on-call rotations. Organizations should avoid patterns where individuals carry on-call responsibilities too frequently, as the stress and sleep disruption associated with after-hours incident response accumulate over time.
Secondary on-call engineers provide backup coverage when primary responders are unavailable or when incidents require additional expertise or hands-on support. Secondary responders should possess senior-level experience and broad technical knowledge, enabling them to quickly understand complex situations and provide effective guidance. The secondary role also serves a mentoring function, with more experienced engineers supporting less experienced primary responders as they develop their incident response capabilities.
Escalation paths define how incidents move up the organizational hierarchy when additional authority, resources, or expertise are required. Effective escalation paths are clearly documented, widely understood, and practiced regularly so that teams don’t fumble with escalation procedures during actual incidents. Escalation should occur based on defined triggers rather than arbitrary time-based rules—for example, escalating to engineering management when incidents exceed defined severity thresholds or duration limits, or when decisions are needed that exceed on-call engineers’ authority.
+-----------------------------------------------------------------------+
| ON-CALL ROTATION STRUCTURE |
+-----------------------------------------------------------------------+
| |
| PRIMARY ON-CALL |
| +-------------------------------------------------------------------+|
| | First responder for all alerts ||
| | Performs initial triage and assessment ||
| | Handles straightforward incidents independently ||
| | Escalates to secondary within 15 minutes when needed ||
| | Maintains incident documentation throughout response ||
| +-------------------------------------------------------------------+|
| | |
| v |
| SECONDARY ON-CALL |
| +-------------------------------------------------------------------+|
| | Provides backup when primary is unavailable ||
| | Assists with complex issues requiring additional expertise ||
| | Offers guidance and mentoring to less experienced primary ||
| | Makes decisions about escalating to management ||
| +-------------------------------------------------------------------+|
| | |
| v |
| ESCALATION PATH |
| +-------------------------------------------------------------------+|
| | Team Lead → Engineering Manager → Director → VP → CTO ||
| | Escalate based on severity, duration, or need for authority ||
| | Executive escalation includes business context and impact ||
| +-------------------------------------------------------------------+|
| |
+-----------------------------------------------------------------------+
Organizations should establish clear expectations about on-call responsibilities, including response time requirements (typically acknowledging pages within five to fifteen minutes), the level of availability expected (must be able to access systems and respond effectively within response time requirements), compensation or time-off policies for on-call duties, and support resources available to on-call engineers. The goal is creating sustainable on-call practices that enable effective incident response without creating excessive burden on individual team members.
Troubleshooting Methodology
Systematic Diagnostic Approach
Effective troubleshooting represents far more than random trial-and-error experimentation. Skilled troubleshooters apply systematic methodologies that efficiently narrow the problem space, test hypotheses in a structured manner, and avoid the common pitfalls of jumping to conclusions or repeatedly investigating dead-end paths. While experienced engineers often appear to troubleshoot intuitively, their effectiveness actually stems from internalized systematic approaches developed through extensive practice.
The troubleshooting process begins with clearly defining the actual problem rather than immediately jumping to solutions. Problem definition involves carefully gathering symptoms from multiple sources, distinguishing between root causes and secondary effects, identifying exactly what functionality is impaired versus what continues to work correctly, and establishing a clear success criterion for resolution. Teams that skip thorough problem definition often waste significant effort solving the wrong problem or implementing solutions that address symptoms rather than root causes.
Once the problem is clearly defined, the next critical step involves determining the scope of impact. Scoping activities identify which users, systems, or geographic regions are affected versus which continue operating normally, determine whether the problem affects all instances of a particular operation or only specific cases, establish the timeline of when the problem began and whether it is getting worse, and identify any patterns in affected versus unaffected cases. This scoping information often provides crucial clues about root causes—for example, problems affecting only a single data center suggest infrastructure issues in that location, while problems affecting only certain user accounts might indicate data corruption or configuration issues.
With clear problem definition and scoping complete, troubleshooters develop hypotheses about potential root causes. Hypothesis generation draws on multiple sources: knowledge of recent changes that might have introduced issues, understanding of how the affected systems operate and what dependencies they rely upon, historical patterns of similar problems and their causes, and analysis of available diagnostic data from logs and monitoring systems. Effective troubleshooters generate multiple competing hypotheses rather than fixating on a single explanation, then systematically test each hypothesis to eliminate impossibilities and confirm actual causes.
Testing hypotheses requires designing and executing experiments that can definitively confirm or eliminate potential causes. Each test should be designed to provide clear, unambiguous results rather than leaving room for continued speculation. When testing potentially disruptive changes, troubleshooters carefully consider the risks and plan appropriate safeguards or rollback procedures. The testing process follows a logical sequence, typically starting with the easiest tests or those most likely to yield useful information, while avoiding tests that risk making the situation worse.
Once testing identifies the root cause, teams implement appropriate solutions. Solution implementation should be deliberate and controlled rather than rushed, with careful attention to potential side effects or unintended consequences. When possible, solutions should be tested in non-production environments before applying to production systems. Teams should also plan rollback procedures before implementing solutions, ensuring they can quickly undo changes if solutions prove ineffective or introduce new problems.
After implementing a solution, verification confirms that the incident is actually resolved. Verification involves testing that the originally impaired functionality now works correctly, confirming that performance and availability meet defined targets, checking that no new problems were introduced by the solution, and obtaining confirmation from users or monitoring systems that service has been restored. Only after thorough verification should incidents be declared resolved.
The final troubleshooting step involves documenting findings, including the problem definition, investigation path, root cause identification, solution implemented, and any lessons learned. This documentation serves multiple purposes: it provides material for postmortem analysis, creates searchable knowledge that can accelerate resolution of similar future issues, demonstrates due diligence for compliance purposes, and contributes to the team’s growing body of operational knowledge.
| Step | Description | Key Activities | Tools and Techniques |
|---|---|---|---|
| 1. Define Problem | Establish exactly what is wrong | Gather symptoms, identify failures, establish success criteria | Monitoring dashboards, user reports, service health checks |
| 2. Determine Scope | Identify what is affected | Map impact across users/systems/locations, identify patterns | Distributed tracing, regional monitoring, user analytics |
| 3. Generate Hypotheses | Develop potential root causes | Review recent changes, analyze symptoms, leverage experience | Change logs, architecture diagrams, historical incidents |
| 4. Test Hypotheses | Systematically verify or eliminate causes | Execute diagnostic tests, gather additional data | CLI tools, log analysis, system metrics, code inspection |
| 5. Implement Solution | Apply appropriate fix | Deploy changes, apply configuration, restart services | IaC tools, deployment pipelines, runbooks |
| 6. Verify Resolution | Confirm problem is solved | Test functionality, check metrics, obtain user confirmation | Synthetic monitoring, health checks, user validation |
| 7. Document Findings | Record investigation and resolution | Write incident timeline, capture lessons learned | Ticketing system, wiki, postmortem documents |
Diagnostic Tools and Techniques
Infrastructure troubleshooting requires proficiency with a wide array of diagnostic tools that provide visibility into system behavior, resource utilization, and network operations. While modern observability platforms provide sophisticated dashboards and visualizations, skilled troubleshooters maintain strong command-line diagnostic skills that enable rapid investigation even when graphical tools are unavailable or when deeper analysis is required.
System-level diagnostics begin with understanding overall resource utilization and performance characteristics. The uptime command provides a quick overview of how long the system has been running and the load average across one, five, and fifteen-minute intervals. Load averages that significantly exceed the number of CPU cores indicate resource contention and performance issues. The top and htop commands provide real-time views of process activity, CPU utilization, memory consumption, and overall system state, enabling quick identification of runaway processes or resource exhaustion.
Memory analysis requires understanding both overall memory availability and how memory is being utilized across different purposes. The free command shows total, used, and available memory, including the critical distinction between memory used for caching (which can be reclaimed if needed) versus memory genuinely consumed by applications. The /proc/meminfo file provides even more detailed memory statistics for advanced analysis. Memory issues often manifest gradually as applications with memory leaks slowly consume available memory until the system reaches a crisis point where the out-of-memory killer begins terminating processes.
Disk subsystem troubleshooting examines both capacity and performance issues. The df command shows filesystem usage, quickly identifying situations where filesystems have filled up and prevented further write operations. The du command helps identify which directories consume the most space when investigating unexpected capacity consumption. For performance analysis, iostat shows detailed I/O statistics including operations per second, throughput, and service times that indicate whether disk subsystems have become bottlenecks. The iotop tool shows real-time I/O by process, identifying which applications are generating excessive disk activity.
Network diagnostics verify connectivity, identify network path issues, and analyze traffic patterns. Basic tools like ping and traceroute confirm reachability and identify where network paths are failing. The ss (socket statistics) and legacy netstat commands show active network connections, listening ports, and socket states that reveal application connectivity issues or port conflicts. The tcpdump tool captures raw network packets for detailed protocol analysis when higher-level tools don’t provide sufficient insight.
Application-level diagnostics examine service-specific logs, trace system calls, and analyze application behavior. The journalctl command provides access to systemd journal logs with powerful filtering and following capabilities. The strace tool traces all system calls made by a process, revealing exactly what operations an application is attempting and where operations are failing or taking excessive time. The lsof (list open files) command shows all files, sockets, and resources opened by processes, helping identify file locks, port bindings, and resource leaks.
# System Overview and Quick Health Check
uptime # Show system uptime and load averages
top -bn1 | head -20 # Process snapshot showing top consumers
vmstat 1 5 # Virtual memory statistics sampled over 5 seconds
iostat -xz 1 5 # Extended disk I/O statistics with zeros suppressed
# CPU Utilization Analysis
mpstat -P ALL 1 5 # Per-CPU utilization statistics
pidstat 1 5 # Per-process resource usage over time
perf top # Real-time CPU profiling showing hot functions
# Memory Analysis and Leak Detection
free -m # Memory usage in megabytes
cat /proc/meminfo # Detailed kernel memory information
ps aux --sort=-%mem | head # Processes sorted by memory consumption
slabtop # Kernel slab cache allocation statistics
# Disk Capacity and Performance
df -h # Human-readable filesystem usage
du -sh /* 2>/dev/null | sort -h # Directory sizes sorted
lsof +D /path # All open files under specific directory
iotop -oa # I/O usage by process, accumulated view
# Network Connectivity and Traffic
ss -tuln # TCP and UDP listening sockets
ss -s # Socket statistics summary
ip addr show # Network interface configuration
tcpdump -i eth0 -n port 80 # Capture HTTP traffic on eth0
# Application-Level Diagnostics
journalctl -u service-name -f # Follow logs for specific systemd service
strace -p PID -c # System call statistics for running process
lsof -p PID # All files and sockets opened by process
pstree -p # Process tree showing parent-child relationships
Effective use of these diagnostic tools requires understanding not just their syntax but also how to interpret their output in context. High CPU utilization might indicate a performance problem or might simply reflect appropriate use of available resources under heavy legitimate load. Network packet loss could indicate infrastructure issues or might result from expected TCP congestion control behavior. Skilled troubleshooters combine data from multiple diagnostic sources to build comprehensive understanding rather than drawing hasty conclusions from single indicators.
Common Infrastructure Failure Patterns
Experience troubleshooting infrastructure incidents reveals recurring patterns that skilled responders learn to recognize quickly. Understanding these common failure patterns enables faster diagnosis by helping troubleshooters quickly narrow the problem space and focus investigation on the most likely causes. While each incident has unique characteristics, many issues fall into predictable categories with well-established diagnostic and resolution approaches.
Resource exhaustion represents one of the most common failure patterns in infrastructure systems. CPU exhaustion manifests as processes competing for available CPU cycles, leading to increased latency as requests queue waiting for CPU availability, potential timeouts as operations take too long to complete, and overall system unresponsiveness in severe cases. CPU exhaustion often results from algorithmic issues in application code, unexpected load increases that exceed system capacity, or runaway processes consuming CPU cycles without useful work. Resolution typically involves identifying and addressing the resource-consuming process, optimizing inefficient operations, or scaling capacity to meet demand.
Memory exhaustion occurs when systems run out of available RAM, forcing increased use of swap space that drastically degrades performance, or triggering the out-of-memory killer that terminates processes to reclaim memory. Memory issues frequently stem from memory leaks in application code where memory is allocated but never properly released, excessive caching that consumes memory without appropriate bounds, or simply insufficient memory allocation for the workload. Resolution requires identifying memory-consuming processes, fixing underlying leaks if present, or increasing available memory.
Disk problems manifest in two distinct forms: capacity exhaustion where filesystems fill up and prevent further writes, and I/O performance issues where disk subsystems cannot keep up with demanded read and write operations. Full filesystems often result from log files growing unbounded, temporary files not being cleaned up properly, or data accumulation outpacing capacity planning. I/O bottlenecks typically result from workload patterns that generate more disk operations than the storage subsystem can handle, poor query patterns that cause excessive database I/O, or hardware failures that degrade storage performance.
Network issues present as connectivity failures where systems cannot reach required destinations, performance degradation where network latency or throughput limits impact application performance, or intermittent failures that manifest as timeout errors affecting some requests while others succeed. Network problems can stem from infrastructure issues such as switch or router failures, configuration errors like incorrect firewall rules or routing tables, DNS resolution failures that prevent hostname lookups, or capacity issues where network links become saturated.
Application-level failures distinct from infrastructure resource issues include configuration errors where incorrect settings prevent proper operation, dependency failures where required external services are unavailable, data corruption issues where invalid data causes processing failures, and software defects where code bugs cause crashes or incorrect behavior. These failures require application-specific troubleshooting approaches that examine application logs, trace execution flow, and analyze application state.
Decision Trees and Troubleshooting Frameworks
Structured troubleshooting frameworks help responders maintain systematic approaches even under the pressure of active incidents. Decision trees provide logical flowcharts that guide investigation through a series of diagnostic questions, with each answer narrowing the problem space and pointing toward specific next steps. While experienced engineers internalize these decision patterns, explicit documentation helps less experienced responders and ensures consistent approaches across the team.
A typical troubleshooting decision tree starts with high-level questions that quickly partition the problem space. The first question often addresses whether recent changes occurred before the problem began—if so, investigating or reverting those changes becomes the immediate focus. When issues correlate with deployments, configuration changes, or infrastructure modifications, the probability that those changes caused or contributed to the problem is sufficiently high to warrant immediate investigation.
If no recent changes are apparent, the investigation shifts to resource-related questions. Is CPU utilization abnormally high? Are memory resources exhausted? Have filesystems filled up? Is disk I/O saturated? Are network connections timing out? Each positive answer leads to specific diagnostic paths that examine the relevant resource subsystem in detail, identify what is consuming the resource, and determine whether the consumption represents legitimate load requiring capacity increases or abnormal behavior requiring remediation.
When resource metrics appear normal, attention turns to dependencies and external factors. Are required external services healthy and responsive? Is database performance degraded? Are third-party APIs returning errors? Are CDN or DNS services functioning properly? Infrastructure systems rarely operate in isolation—they depend on numerous external components whose failures can manifest as apparent issues in the infrastructure being investigated.
+-----------------------------------------------------------------------+
| TROUBLESHOOTING DECISION TREE |
+-----------------------------------------------------------------------+
| |
| [Service Degraded] |
| | |
| v |
| [Recent Changes?] |
| / \ |
| Yes No |
| / \ |
| v v |
| [Can Rollback?] [Resource Issue?] |
| / \ / \ |
| Yes No Yes No |
| / \ / \ |
| v v v v |
| [Execute [Investigate [Scale or [Dependency |
| Rollback] Change Optimize] Issue?] |
| Impact] | |
| [Yes/No] |
| / \ |
| v v |
| [Investigate [Deep |
| Dependencies] Diagnostic |
| Analysis] |
| |
+-----------------------------------------------------------------------+
The decision tree framework helps prevent common troubleshooting mistakes such as prematurely concluding that an issue is resolved when symptoms temporarily improve without addressing root causes, investigating the same areas repeatedly when initial investigation found no issues, or fixating on a particular hypothesis and failing to consider alternatives when evidence doesn’t support the initial theory. By following structured decision paths, troubleshooters maintain disciplined investigation approaches that systematically eliminate possibilities and converge on actual root causes.
Root Cause Analysis
Understanding Root Cause Versus Proximate Cause
Effective incident management distinguishes between proximate causes—the immediate factors that directly triggered an incident—and root causes, the underlying systemic issues that created conditions where the incident could occur. Proximate causes are typically easy to identify and address, but resolving only proximate causes leaves organizations vulnerable to similar incidents recurring under slightly different circumstances. True root cause analysis digs deeper to understand why the proximate cause was possible and what systemic improvements would prevent entire categories of similar issues.
Consider an incident where a production database failure caused a major application outage. The proximate cause might be identified as “database server ran out of disk space.” However, this proximate cause explanation immediately raises deeper questions: Why did the database run out of space? Was data growing faster than expected? Were logs not being rotated properly? Why didn’t monitoring alert the team before space was completely exhausted? Why wasn’t there automatic space management? Each of these questions potentially reveals root causes that, if addressed, would prevent similar incidents.
Root cause analysis often reveals multiple contributing factors rather than a single definitive cause. Modern complex systems exhibit behavior that emerges from the interaction of multiple components, configurations, and operational practices. When incidents occur, they typically result from the alignment of several conditions that individually might be manageable but in combination produce failure. Effective root cause analysis identifies all significant contributing factors, recognizing that addressing only one factor might not be sufficient to prevent recurrence.
The distinction between “root cause” and contributing factors can be subtle and is sometimes debated even among incident analysts. In practice, teams should focus less on definitively identifying “the” root cause and more on identifying all factors that, if addressed, would make similar incidents less likely or less severe. This pragmatic approach ensures that root cause analysis drives meaningful improvements rather than becoming an academic exercise in identifying the most fundamental cause.
Root Cause Analysis Techniques
Multiple structured techniques exist for conducting root cause analysis, each offering different perspectives and strengths for different types of incidents. Organizations often employ multiple techniques for complex incidents, using each method to illuminate different aspects of the situation and cross-validate findings.
The Five Whys technique provides a simple but powerful method for drilling down from proximate causes to underlying root causes. Starting with the observed problem, analysts repeatedly ask “why did this happen?” for each answer, typically requiring five iterations (though sometimes fewer or more) to reach root causes. Each “why” question should be answered with concrete, specific responses rather than vague generalities. The power of Five Whys lies in its simplicity and its ability to prevent teams from stopping their analysis too early.
A Five Whys analysis of a production outage might proceed as follows: Problem: Production API returned errors for thirty minutes. Why did the API return errors? Because the database connection pool was exhausted. Why was the connection pool exhausted? Because database connections were not being properly released. Why were connections not being released? Because a code change introduced a connection leak under specific conditions. Why wasn’t the leak caught before production? Because the change was not load tested. Why wasn’t load testing performed? Because the deployment process didn’t require load testing for this type of change. This analysis reveals that while the proximate cause was exhausted database connections, the root cause involved missing process requirements around load testing.
+-----------------------------------------------------------------------+
| 5 WHYS ANALYSIS EXAMPLE |
+-----------------------------------------------------------------------+
| |
| PROBLEM: Production API returned 500 errors for 30 minutes |
| |
| WHY 1: Why did the API return 500 errors? |
| → The database connection pool was exhausted, preventing |
| the application from executing queries |
| |
| WHY 2: Why was the connection pool exhausted? |
| → Connections were being opened but not properly released |
| back to the pool after use |
| |
| WHY 3: Why were connections not being released? |
| → A code change introduced a connection leak in the |
| error handling path |
| |
| WHY 4: Why wasn't the connection leak caught before production? |
| → The change underwent code review and unit testing but |
| no load testing was performed |
| |
| WHY 5: Why wasn't load testing performed? |
| → The deployment process categorized this as a "small fix" |
| not requiring comprehensive load testing |
| |
| ROOT CAUSE: Deployment process lacks adequate criteria for when |
| load testing is required, allowing changes with potential |
| performance impact to skip this critical validation |
| |
| CORRECTIVE ACTIONS: |
| 1. Add mandatory load testing for all database-touching code |
| 2. Implement connection pool monitoring with alerting |
| 3. Add static analysis to detect potential connection leaks |
| 4. Update code review guidelines to explicitly check resource mgmt |
| |
+-----------------------------------------------------------------------+
Fishbone diagrams (also called Ishikawa diagrams or cause-and-effect diagrams) provide a visual method for exploring multiple categories of potential contributing factors. The diagram places the problem statement at the “head” of the fish, with major category branches extending from the spine and specific factors listed along each branch. Traditional categories include People (human factors like training, staffing, communication), Process (procedures, workflows, approvals), Technology (hardware, software, tools), Environment (external factors, dependencies, physical conditions), Measurement (monitoring, metrics, visibility), and Materials (data, configuration, third-party components).
Fishbone diagrams excel at ensuring comprehensive analysis that considers multiple perspectives rather than fixating on a single category of causes. They also provide excellent visual communication of complex incident causation, making them valuable for presenting findings to stakeholders. When conducting fishbone analysis, teams brainstorm potential contributing factors in each category, then analyze which factors actually contributed to the specific incident and which represent theoretical concerns that weren’t relevant in this case.
+-----------------------------------------------------------------------+
| FISHBONE DIAGRAM |
+-----------------------------------------------------------------------+
| |
| PEOPLE PROCESS TECHNOLOGY |
| / | \ |
| - Insufficient - Missing - Connection pool |
| training on load testing too small for |
| connection mgmt requirement peak load |
| - On-call lacks - No resource - No automated |
| DB expertise review in CR connection leak |
| - Fast-track detection |
| \ for "small" / | |
| \ changes / | |
| v v v |
| +----------------------------+ |
| | DATABASE OUTAGE | |
| +----------------------------+ |
| ^ ^ ^ |
| / | \ |
| / | \ |
| ENVIRONMENT MEASUREMENT MATERIALS |
| | | \ |
| - Traffic spike - No connection - Old JDBC driver |
| from campaign pool monitoring with known leak |
| - Database in - Alert fatigue - Config didn't |
| degraded state masked warnings match prod load |
| |
+-----------------------------------------------------------------------+
Timeline analysis arranges all relevant events in chronological order to understand incident progression and identify critical decision points or missed opportunities for earlier intervention. This technique is particularly valuable for complex incidents involving multiple systems or teams, where understanding the sequence and interaction of events is crucial. Timeline analysis often reveals that what initially appeared as a single incident actually represents a cascade of multiple smaller issues, or identifies the specific moment when the incident became inevitable barring extraordinary intervention.
Fault tree analysis uses logic diagrams to map out how various component failures could combine to produce system-level failures. This technique, borrowed from reliability engineering, helps identify single points of failure, understand redundancy requirements, and assess the effectiveness of proposed improvements. Fault trees work backward from the undesired event (the top event) through intermediate events connected by logical gates (AND, OR) down to basic events that represent component failures. This systematic mapping reveals which combinations of component failures lead to system failures and helps prioritize improvements based on their effectiveness in preventing failure scenarios.
Blameless Postmortem Culture
The cultural approach to postmortem analysis profoundly influences organizational learning and continuous improvement. Blameless postmortem culture recognizes that individuals operating within complex systems rarely cause incidents through malice or gross negligence—instead, incidents typically result from reasonable decisions made with incomplete information, systemic issues that created error-prone conditions, or the emergent behavior of complex systems that no individual fully understands.
Blameless culture does not mean accountability-free culture. Rather, it shifts accountability from individual actions to systemic conditions and organizational practices. When an engineer deploys a change that causes an outage, blameless culture asks not “why did this engineer make a mistake?” but rather “what systemic factors made this type of error possible?” and “what organizational improvements would make similar errors less likely?” This shift in perspective from individual blame to systemic learning creates psychological safety that encourages honest disclosure of mistakes, detailed analysis of what went wrong, and open discussion of improvement opportunities.
Organizations that maintain blame culture—where incidents trigger searches for culprits and individuals face punitive consequences for mistakes—inevitably drive incident information underground. Engineers learn to hide mistakes, minimize incident severity, and avoid the detailed documentation that would support learning. The short-term consequence is that management loses visibility into operational problems. The long-term consequence is that the organization fails to learn from incidents and repeatedly experiences similar issues.
Effective blameless postmortems follow structured formats that ensure comprehensive documentation while maintaining focus on learning rather than blame. The postmortem document typically includes an executive summary providing a brief description of what happened and the overall impact, a detailed timeline documenting all relevant events with precise timestamps, impact analysis quantifying business and technical consequences, root cause identification explaining underlying issues rather than just proximate causes, contributing factors that made the incident more likely or more severe, and action items with clear ownership and deadlines.
Postmortem Documentation and Action Items
Postmortem documents serve multiple audiences with different needs: executives need brief summaries of business impact and high-level corrective actions, technical teams need detailed analysis that supports their learning and improvement efforts, and audit or compliance functions may need evidence of due diligence and process adherence. Effective postmortem documents provide appropriate information at different levels of detail through careful structure and writing.
# Incident Postmortem: Database Connection Pool Exhaustion
## Executive Summary
On January 15, 2024, at 14:23 UTC, the production API experienced a complete outage lasting 22 minutes, affecting approximately 10,000 users and resulting in estimated revenue impact of $50,000. The incident was caused by database connection pool exhaustion introduced by a code change deployed that morning. The system was restored via rollback to the previous version. Five action items have been identified to prevent similar incidents, including enhanced monitoring, expanded load testing requirements, and improved deployment processes.
## Detailed Timeline (All times UTC)
| Time | Event | Person/System |
|------|-------|---------------|
| 09:15 | v2.3.1 deployment initiated via CI/CD pipeline | Jenkins |
| 09:30 | Deployment completed successfully to all production instances | DevOps Team |
| 14:20 | Traffic begins increasing due to scheduled marketing campaign | Marketing |
| 14:23 | Alert: API error rate exceeds 1% threshold | Monitoring System |
| 14:25 | On-call engineer acknowledges alert, begins investigation | Engineer A |
| 14:28 | Database connection pool metrics examined, showing 100% utilization | Engineer A |
| 14:30 | Incident declared P1, incident commander assigned | Commander B |
| 14:32 | Recent deployments identified as investigation focus | Engineer A |
| 14:35 | Code review of v2.3.1 identifies potential connection leak in error path | Engineer A |
| 14:37 | Decision made to rollback to v2.3.0 | Commander B |
| 14:38 | Rollback initiated | DevOps Team |
| 14:45 | Rollback completed, connection pool utilization returns to normal | System |
| 14:47 | Error rate drops below threshold, service restored | System |
| 15:00 | Monitoring confirms stable operation, incident closed | Commander B |
| 15:30 | Postmortem scheduled for January 16 at 10:00 UTC | Commander B |
## Impact Analysis
**Duration:** 22 minutes of complete service outage plus 10 minutes of degraded service during rollback
**Users Affected:** Approximately 10,000 active users during the outage window
**Revenue Impact:** Estimated $50,000 based on average transaction value and volume during affected period
**SLA Impact:** 22 minutes of downtime represents 0.05% of monthly availability, consuming 5% of the monthly error budget
**Customer Support:** 127 support tickets filed, requiring approximately 40 hours of support team time
**Reputational Impact:** 23 social media mentions, generally understanding given rapid resolution and transparent communication
## Root Cause Analysis
The immediate cause of the outage was exhaustion of the database connection pool. Under normal conditions, the application acquires database connections from the pool for query execution and releases them back to the pool when operations complete. Version 2.3.1 introduced a subtle bug in error handling code where exceptions occurring during transaction commit resulted in connections not being properly released.
Under low to moderate load, the connection leak rate was sufficiently slow that the periodic pool cleanup mechanisms maintained adequate available connections. However, when traffic increased due to the scheduled marketing campaign, the higher query volume increased the rate of connection acquisition and thus the rate of leakage. Within 20 minutes of the traffic increase, all available connections were exhausted, causing all subsequent requests to fail with connection timeout errors.
The root cause extends beyond the code defect itself to several systemic issues:
1. **Inadequate testing requirements:** The change was categorized as a "small fix" and underwent only unit testing and manual validation. No load testing was performed that would have revealed the connection leak under sustained traffic.
2. **Insufficient monitoring:** While the system had monitoring for overall database performance and API error rates, no specific monitoring existed for connection pool utilization that would have provided early warning as available connections depleted.
3. **Deployment timing:** The deployment occurred during business hours on the same day as a scheduled marketing campaign, creating an unnecessarily short window between deployment and traffic increase.
4. **Code review gaps:** While the change underwent code review, reviewers did not identify the connection leak, suggesting that review guidelines don't adequately emphasize resource management patterns.
## Contributing Factors
Several factors contributed to making this incident possible or more severe:
1. The connection pool size had not been adjusted despite recent traffic growth, leaving less margin for abnormal conditions
2. The JDBC driver version in use was two releases behind current, with known issues related to connection management in error scenarios
3. Recent alert tuning had adjusted thresholds for database-related alerts, potentially delaying detection
4. On-call rotation schedule meant the engineer responding had less experience with database-related issues
5. The rollback process, while ultimately successful, required manual approval steps that added several minutes to resolution time
## What Went Well
Despite the incident severity, several aspects of the response were effective:
1. Monitoring detected the issue within 3 minutes of error rates increasing
2. The on-call engineer responded promptly and followed systematic troubleshooting procedures
3. Incident command structure was activated quickly and roles were clearly understood
4. Communication to stakeholders was timely and transparent
5. Rollback capability existed and executed successfully
6. Comprehensive logs enabled post-incident analysis
## Action Items
| Action | Owner | Due Date | Priority | Status |
|--------|-------|----------|----------|--------|
| Add connection pool monitoring with alerting at 80% and 90% utilization | Platform Team | Jan 20 | P0 | Complete |
| Implement automated load testing in CI/CD for all database-related changes | DevOps Team | Jan 25 | P0 | In Progress |
| Update deployment process to prohibit deployments within 24h of scheduled campaigns | Release Mgmt | Jan 22 | P1 | Complete |
| Upgrade JDBC driver to current version and test thoroughly | Backend Team | Jan 30 | P1 | Todo |
| Review and update code review guidelines to emphasize resource management | Engineering | Feb 5 | P2 | Todo |
| Implement static analysis rules to detect potential connection leaks | Platform Team | Feb 10 | P2 | Todo |
| Increase connection pool size to accommodate peak traffic patterns | DBA Team | Jan 18 | P1 | Complete |
| Automate rollback approval for P1 incidents | DevOps Team | Feb 15 | P2 | Todo |
| Create runbook for database connection pool issues | On-Call Team | Jan 25 | P2 | In Progress |
| Schedule training on database connection management patterns | Backend Team | Mar 1 | P3 | Todo |
## Lessons Learned
1. **Resource management requires explicit testing:** Connection pools, thread pools, file handles, and similar resources must be explicitly tested under load conditions that simulate production traffic patterns
2. **Categorization schemes can mislead:** Classifying changes as "small" or "low risk" based on surface characteristics (lines of code changed, files modified) can miss fundamental risks related to what those changes actually do
3. **Monitoring gaps create blind spots:** Lack of monitoring for connection pool utilization meant the team had no visibility into the resource depletion until failures manifested in user-visible errors
4. **Timing matters:** Deploying changes immediately before known traffic increases leaves minimal time for issues to manifest under normal conditions and compounds incident severity
5. **Defense in depth:** Multiple layers of protection (testing, monitoring, capacity margins, rapid rollback) are necessary because no single layer is perfect
## References
- Incident ticket: INC-2024-0115-001
- Deployment log: https://jenkins.example.com/job/api-deploy/build/543
- Monitoring dashboards: https://grafana.example.com/d/api-health
- Code change: https://github.com/example/api/pull/1234
Effective postmortem processes include follow-up mechanisms that ensure action items actually get completed rather than languishing indefinitely. Common approaches include designating an action item owner who tracks progress and reports status, scheduling a 30-day review to assess completion and effectiveness of implemented changes, and incorporating action item status into team metrics and management reporting. Without deliberate follow-through mechanisms, even thoughtful postmortems fail to drive lasting improvement.
Runbooks and Operational Documentation
Purpose and Structure of Runbooks
Runbooks represent formalized operational knowledge that enables consistent, rapid response to known classes of incidents. While experienced engineers build mental models of how to handle various problems through repeated exposure, runbooks externalize this knowledge, making it accessible to less experienced team members, maintaining consistency across the team, and preserving institutional knowledge even as team membership changes.
Effective runbooks strike a careful balance between comprehensiveness and usability. Overly brief runbooks that simply list commands without context leave responders confused about when each step is appropriate or what outcomes to expect. Conversely, excessively detailed runbooks that attempt to cover every conceivable variation become unwieldy documents that no one actually follows during high-pressure incidents. The goal is providing sufficient information for a moderately experienced engineer to handle the situation effectively while maintaining a clear, scannable structure that supports rapid access to needed information.
Runbook structure typically follows a consistent template that matches how responders actually use the documentation. The runbook begins with an overview that clearly identifies what situation the runbook addresses, enabling responders to quickly determine whether they’ve selected the correct runbook. This overview includes the specific symptoms or alert names that indicate the runbook applies, helping on-call engineers rapidly navigate from an alert to the appropriate response procedure.
Prerequisites section lists required access permissions, tools, credentials, or other preparation needed before executing the runbook procedures. This section prevents frustrating situations where responders begin following a runbook only to discover halfway through that they lack necessary access or tools. By clearly stating prerequisites upfront, runbooks enable responders to either confirm they have everything needed or quickly engage assistance from those who do.
Diagnosis steps guide responders through confirming that the situation actually matches the runbook’s scenario and understanding the specific nature of the issue. These steps prevent the common mistake of applying solutions without fully understanding the problem, which can waste time or even make situations worse. Diagnosis steps typically include specific commands to run, metrics to check, or logs to examine, along with guidance about what different outcomes indicate about the nature of the problem.
Resolution steps provide clear, sequential instructions for addressing the confirmed issue. These steps are written as executable procedures that responders can follow systematically, including specific commands with example outputs that help responders confirm they’re achieving expected results at each stage. When multiple resolution paths exist (for example, different approaches depending on whether the issue is load-based versus process-based), the runbook uses clear decision points that guide responders to the appropriate path.
Verification steps ensure that implemented solutions actually resolved the incident. These steps specify exactly what should be checked to confirm resolution, including specific metrics that should return to normal ranges, operations that should now succeed, or user confirmations that should be obtained. Without explicit verification steps, runbooks risk situations where responders prematurely declare incidents resolved only to have them recur shortly afterward.
Escalation guidance specifies when and how to escalate beyond runbook procedures. Clear escalation criteria prevent responders from feeling they must exhaust every possible option in the runbook before seeking help, while also preventing premature escalation before readily available solutions have been attempted. Escalation sections identify who to contact, what information to provide when escalating, and what circumstances absolutely require immediate escalation.
Sample Runbook: High CPU Utilization
The following example demonstrates practical runbook structure for a common infrastructure scenario:
# Runbook: High CPU Utilization
## Overview
This runbook addresses alerts for high CPU utilization on production application servers. High CPU conditions can cause increased latency, timeouts, and degraded user experience. This runbook helps diagnose the cause and implement appropriate remediation.
## Alert Triggers
- **HighCPUUtilization:** CPU usage > 85% for 10+ minutes on any production instance
- **CriticalCPUUtilization:** CPU usage > 95% for 5+ minutes on any production instance
## Symptoms
- Slow application response times (request latency increased 2-5x normal)
- Elevated timeout errors in application logs
- Alert notification: "High CPU utilization on "
- User reports of slow page load times
## Prerequisites
- SSH access to affected server(s)
- Sudo/root privileges on production instances
- Access to monitoring dashboards at https://monitoring.example.com
- Access to deployment logs at https://deploy.example.com
- PagerDuty incident should already be created by alerting system
## Diagnosis Steps
### Step 1: Verify Current CPU State
First, confirm that the alert reflects current reality rather than a stale condition:
```bash
# Check current overall CPU utilization
uptime
top -bn1 | head -10
# Expected output shows load averages and top processes
# If load average is below 0.70 per CPU, alert may be stale
# (Example: 8-core system should see load < 5.6 for normal conditions)
If current CPU utilization is normal, verify the alert timestamp and check monitoring dashboard to confirm the spike has passed. If spike has resolved on its own, proceed to Step 3 to understand what occurred.
Step 2: Identify High CPU Consumers
Determine which process(es) are consuming CPU resources:
# List processes sorted by CPU utilization
ps aux --sort=-%cpu | head -20
# For processes using > 50% CPU, get additional detail
top -bn1 -p <PID>
# Check process command line and runtime
ps -p <PID> -o pid,ppid,cmd,%cpu,%mem,etime,user
Common findings and interpretations:
- Application process using 80-95% CPU: Likely handling legitimate load or executing expensive operation
- Single process at 100% CPU for extended time: Potential runaway process or infinite loop
- Many processes each using moderate CPU: Overall system load issue, likely legitimate traffic
- System processes (kswapd, etc.) using high CPU: Possible memory pressure or I/O bottleneck
Step 3: Check Recent Changes
Correlation with recent changes often provides critical diagnostic information:
# Check recent deployments (last 4 hours)
curl -s https://deploy.example.com/api/recent | jq '.deployments[] | select(.timestamp > '$(date -d '4 hours ago' +%s)')'
# Check for recent configuration changes
sudo journalctl --since "4 hours ago" | grep -i "config"
# Verify no unusual cron jobs running
sudo ps aux | grep cron
crontab -l # Check user crontab
sudo crontab -l # Check root crontab
If a recent deployment correlates with the CPU spike, proceed to Resolution Option A (Process-Specific Issue) and consider rollback. If no changes are evident, continue to Step 4.
Step 4: Assess Traffic Patterns
Determine if CPU elevation results from increased legitimate load:
# Check current request rate (adjust for your metrics endpoint)
curl -s localhost:9090/metrics | grep http_requests_total
# Compare to historical average
curl -s 'https://monitoring.example.com/api/query?query=rate(http_requests_total[5m])'
# Check if request patterns show unusual distribution
curl -s localhost:9090/metrics | grep http_request_duration
If traffic is significantly elevated but request patterns are normal, this indicates legitimate load requiring capacity adjustment (proceed to Resolution Option B). If request patterns are unusual or suspicious, consider whether the traffic is legitimate before adding capacity.
Resolution Procedures
Option A: Process-Specific Issue
Use when a specific process is consuming excessive CPU due to application-level issues:
# 1. Document the process state for later analysis
ps -p <PID> -f > /tmp/high_cpu_process_$PID.txt
sudo lsof -p <PID> > /tmp/high_cpu_process_${PID}_files.txt
# 2. Check if process is critical application service
systemctl status <service-name>
# 3. If this is the main application process and a recent deployment occurred:
# Consider rollback if within 2 hours of deployment
# Get current version
curl -s http://localhost/version
# Initiate rollback via deployment tool
# (Requires incident commander approval for production)
./deploy/rollback.sh --environment=production --version=<previous_version>
# 4. If not appropriate for rollback, restart the specific service:
sudo systemctl restart <service-name>
# 5. Monitor recovery over the next 5-10 minutes
watch -n 5 'ps aux --sort=-%cpu | head -10'
# Verify application health after restart
curl -s http://localhost/health | jq
Verification:
- CPU utilization returns below 75% within 5 minutes
- Application health checks return success (200 OK)
- Request latency returns to normal (p95 < 500ms)
- No error rate increase following restart
Option B: Load-Based Issue
Use when overall system load is high due to increased legitimate traffic:
# 1. Verify this is actually legitimate load, not an attack
# Check for unusual request patterns, geographic anomalies, etc.
curl -s 'https://monitoring.example.com/api/security/traffic_analysis'
# 2. Confirm auto-scaling is functioning
aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names production-web \
--query 'AutoScalingGroups[0].[DesiredCapacity,MinSize,MaxSize]'
# 3. If auto-scaling is at maximum, consider temporary increase
# (Requires approval from on-call manager for cost implications)
# Increase max size temporarily
aws autoscaling update-auto-scaling-group \
--auto-scaling-group-name production-web \
--max-size 20
# 4. If auto-scaling is not enabled or not responding, manually scale:
aws autoscaling set-desired-capacity \
--auto-scaling-group-name production-web \
--desired-capacity <NEW_CAPACITY>
# 5. For immediate relief, consider enabling traffic shedding
# (if available in load balancer)
# This allows system to reject some requests rather than timing them all out
# Example: Configure ALB to shed 20% of traffic
aws elbv2 modify-target-group \
--target-group-arn <TG_ARN> \
--health-check-interval-seconds 10 \
--unhealthy-threshold-count 2
Verification:
- CPU utilization drops below 80% within 10 minutes
- Request success rate returns above 99%
- p95 latency returns below SLA threshold
- Auto-scaling policy adjusts capacity based on demand
Option C: Runaway Process
Use for clearly broken processes consuming resources without useful work:
# 1. Capture diagnostic information before killing process
ps -p <PID> -o pid,ppid,cmd,%cpu,%mem,etime > /tmp/runaway_${PID}.txt
sudo strace -p <PID> -c -f -o /tmp/strace_${PID}.txt &
sleep 10 # Collect 10 seconds of system calls
sudo kill -STOP <PID> # Stop strace
# 2. For Java processes, capture thread dump
sudo kill -3 <PID> # SIGQUIT to Java process
# Thread dump will appear in application logs
# 3. Attempt graceful termination first
sudo kill -15 <PID> # SIGTERM
sleep 30 # Allow time for graceful shutdown
# 4. Verify process is gone
ps -p <PID>
# 5. If process is still running, force kill
sudo kill -9 <PID> # SIGKILL
# 6. If this was a managed service, restart it
sudo systemctl start <service-name>
Verification:
- Runaway process no longer appears in process list
- CPU utilization drops to normal range
- If process was restarted, verify it’s running correctly
- Application functionality is restored
Post-Resolution Activities
Document the Incident
After resolving the immediate issue, document what occurred:
- Update the PagerDuty incident with resolution details
- Add timeline to incident tracking system
- Note which resolution option was used and why
- Capture any relevant metrics or logs for postmortem
Monitor for Recurrence
Continue monitoring for 30-60 minutes after resolution:
# Watch CPU over time
watch -n 10 'uptime; echo "---"; ps aux --sort=-%cpu | head -5'
# Monitor application metrics
watch -n 10 'curl -s localhost:9090/metrics | grep -E "cpu|latency|error"'
Set a reminder to check again in 4-8 hours to confirm the issue hasn’t returned.
Escalation
Escalate immediately if:
- Issue persists after 30 minutes despite following this runbook
- Root cause is unclear and requires deeper application or infrastructure knowledge
- Multiple servers are affected simultaneously (potential systemic issue)
- CPU issue correlates with other alerts (memory, disk, network)
- Security implications are suspected
Escalation Path:
- Secondary on-call: @platform-secondary
- Team Lead: @platform-lead
- Engineering Manager: @eng-manager
Escalation Channel: #incident-response Slack channel
Include in Escalation:
- Incident ticket number
- Affected instance(s)
- What you’ve tried so far
- Current impact to users/services
- Any relevant metrics or log excerpts
Related Runbooks
- High Memory Utilization
- Application Restart Procedures
- Emergency Rollback Process
- Auto-scaling Troubleshooting
Revision History
| Date | Changes | Author | |——|———|——–| | 2024-01-15 | Initial version | Platform Team | | 2024-01-22 | Added auto-scaling details | DevOps Team | | 2024-02-05 | Updated escalation path | On-Call Lead |
### Maintaining Runbook Currency
Runbooks represent living documentation that must evolve as systems, processes, and organizational knowledge change. Outdated runbooks are often worse than no runbooks at all, as they lead responders down wrong paths while creating false confidence that they're following correct procedures. Maintaining runbook currency requires deliberate practices integrated into operational workflows.
Each time a runbook is used during an actual incident, the responder should note any discrepancies between documented procedures and actual reality. Were commands listed in the runbook accurate? Did expected outputs match actual outputs? Were there missing steps that the responder had to figure out? Were there steps that were no longer necessary or relevant? These notes should be captured in the incident documentation and then used to update the runbook.
Runbooks should undergo scheduled reviews on a regular cadence even when they haven't been used recently. Quarterly reviews work well for frequently-used runbooks, while annual reviews may suffice for rarely-needed procedures. Reviews should verify that documented commands still work, that referenced systems and tools still exist at the specified locations, that escalation contacts are still current, and that any dependencies or prerequisites haven't changed.
When infrastructure changes are implemented—new tools adopted, systems migrated, architectures refactored—teams should explicitly identify which runbooks are affected and update them as part of the change process. Including runbook updates in change checklists helps prevent the common problem where systems evolve but documentation lags behind, gradually becoming less and less accurate.
---
## Communication During Incidents
### Stakeholder Communication Strategies
Effective incident response requires not only technical problem-solving but also clear, timely communication with various stakeholders who have different information needs and different levels of technical expertise. Communication strategy must consider multiple audiences: technical team members actively responding to the incident need detailed technical information and real-time coordination; executives need business-context information about impact and expected resolution timeframe; customers need transparency about what they're experiencing and when service will be restored; and support teams need information to help them respond to customer inquiries.
Communication during incidents follows a predictable lifecycle that mirrors the incident response lifecycle itself. The initial detection phase triggers internal technical communication through alerting systems and chat channels, bringing responders together and establishing shared awareness of the problem. As the incident moves into triage and initial assessment, communication expands to include incident command structure formation and initial stakeholder notification for high-severity incidents.
During the active response phase, communication focuses on coordination among responders and regular status updates to stakeholders. The frequency and format of updates should be established at the beginning of the incident and then maintained consistently. For critical incidents, updates every fifteen to thirty minutes keep stakeholders informed even when there's no major progress to report—the absence of information during an incident creates anxiety and speculation that is often worse than the reality. Updates should be factual and transparent, clearly distinguishing between what is known, what is suspected but unconfirmed, and what remains uncertain.
Status updates should follow a consistent structure that provides key information efficiently: current situation description (what is happening right now), impact summary (who and what is affected), actions being taken (what the team is doing to resolve the issue), next steps (what will happen next), and estimated time to resolution or next update (setting expectations about when stakeholders will hear more). When time estimates are provided, they should be realistic and include appropriate caveats—it's better to provide a conservative estimate and resolve faster than expected than to provide an aggressive estimate and miss it.
As the incident moves toward resolution, communication shifts to verification announcements and final resolution notices. Resolution communications should acknowledge the impact experienced, thank teams and individuals involved in response, provide appropriate apology for customer impact, and indicate when and where more detailed postmortem information will be available. For significant incidents, follow-up communication should occur after initial resolution to provide comprehensive incident analysis and explain what improvements are being implemented.
### Internal Communication Channels
Modern incident response relies heavily on real-time communication tools that enable rapid information sharing and coordination across distributed teams. Dedicated incident channels—typically implemented in tools like Slack, Microsoft Teams, or similar platforms—provide focused communication spaces where all incident-related discussion occurs, reducing noise and making it easier for team members to maintain situational awareness. These channels should be created automatically when significant incidents are declared, following consistent naming conventions that make them easy to identify.
Incident channels serve multiple purposes beyond just real-time discussion. They create a persistent record of incident response activities that proves invaluable during postmortem analysis. They provide a space where the incident scribe can post timeline entries that document key events. They enable asynchronous participation, allowing team members to review channel history and understand the current situation even if they weren't present when the incident began. They facilitate collaboration with subject matter experts who can be pulled into the channel as their expertise is needed.
Effective use of incident channels requires some discipline to prevent them from becoming chaotic. Many teams establish conventions such as using threaded replies for detailed technical discussion that's not essential for everyone to follow in real-time, posting major updates as top-level messages so they're immediately visible to anyone scanning the channel, using specific emoji reactions to indicate actions taken or information acknowledged, and having the incident commander or scribe periodically post concise status summaries that help everyone maintain shared understanding.
Video conferencing or voice channels complement text-based communication during major incidents, enabling richer real-time collaboration when teams need to discuss complex technical details or coordinate multiple parallel response activities. Some teams maintain persistent incident response voice channels that responders automatically join when significant incidents occur, providing immediate voice coordination capability without requiring setup during the high-pressure incident initiation.
### External Communication and Status Pages
Public-facing communication during incidents requires careful attention to transparency, accuracy, and appropriate business language. Status pages—either standalone services or integrated into company websites—serve as the authoritative source of truth about service availability and current incidents. Effective status page communication balances providing useful information with avoiding technical details that may confuse non-technical audiences or expose security-relevant information.
Initial incident notification on status pages should occur promptly when user-facing impact is confirmed, typically within the first ten to fifteen minutes of incident detection. This initial notice acknowledges that the organization is aware of issues, validates that user-experienced problems reflect real service issues rather than isolated cases, and demonstrates organizational responsiveness. The initial update should be concise and factual, avoiding speculation about causes or resolution timelines when investigation is just beginning.
[INVESTIGATING] Elevated Error Rates on API Service We are investigating reports of elevated error rates affecting our API service. Some users may experience slow responses or intermittent errors. Our engineering team is actively investigating the cause and working toward resolution.
Posted: 14:25 UTC
Progress updates maintain transparency and manage expectations during extended incidents. These updates should occur at regular intervals—typically every thirty to sixty minutes for major incidents, potentially less frequently for lower-severity issues. Even when investigation is ongoing without major breakthroughs, updates that acknowledge continued work help prevent speculation and demonstrate that the incident hasn't been forgotten. Updates should revise the status tag as understanding evolves: INVESTIGATING when the issue is being diagnosed, IDENTIFIED once root cause is understood, and MONITORING during recovery verification.
[IDENTIFIED] Elevated Error Rates on API Service We have identified the cause as a database connection management issue introduced in this morning’s deployment. Our engineering team is implementing a fix. We expect service to be restored within the next 30 minutes. We apologize for the disruption.
Posted: 14:40 UTC
Resolution announcements confirm that service has been restored and the incident is closed. These messages should be factual and appropriately apologetic without being overly dramatic. Include the incident duration to provide context about impact, briefly describe the resolution without unnecessary technical detail, confirm that service is now operating normally, and commit to follow-up analysis for significant incidents.
[RESOLVED] Elevated Error Rates on API Service The issue has been resolved through a rollback to the previous application version. A database connection pool exhaustion issue was identified and addressed. All services are now operating normally. The incident lasted approximately 22 minutes.
We apologize for any inconvenience this incident caused. We are conducting a full analysis to prevent similar issues in the future.
Posted: 15:00 UTC ```
Review Questions
Describe the key phases of the incident lifecycle and explain how each phase contributes to effective incident management. What happens if teams skip directly from detection to response without adequate triage?
Explain how incident priority should be determined and why organizations need clear classification schemes. What are the risks of either over-prioritizing or under-prioritizing incidents?
What is the difference between a proximate cause and a root cause? Why is it important to identify root causes rather than stopping analysis at proximate causes?
Describe the key elements of a blameless postmortem culture. How does blameless culture differ from accountability-free culture?
What sections should a comprehensive runbook contain, and what purpose does each section serve? How do runbooks differ from general operational documentation?
Why is regular communication important during incidents, even when there’s no significant progress to report? What are the risks of communication gaps during incident response?
Describe the systematic troubleshooting methodology and explain how it differs from trial-and-error approaches. What are the risks of unsystematic troubleshooting?
What is the role of an incident commander, and what skills are most important for this role? Why is it important to separate the incident commander role from the technical lead role during major incidents?
Key Takeaways
Structured incident management processes enable organizations to respond consistently and effectively to infrastructure issues, minimizing both the duration and severity of service disruptions through systematic detection, triage, response, and recovery activities.
Clear role definitions and incident command structure prevent confusion during high-pressure situations, ensuring that all necessary activities receive attention while avoiding duplication of effort or gaps in coverage that could prolong incidents.
Systematic troubleshooting methodologies dramatically improve diagnostic efficiency by helping responders methodically narrow the problem space through hypothesis formation and testing rather than relying on random trial-and-error approaches.
Root cause analysis that goes beyond proximate causes drives meaningful systemic improvements by identifying underlying issues that, when addressed, prevent entire categories of similar incidents rather than just addressing individual symptoms.
Blameless postmortem culture creates psychological safety that enables honest disclosure and thorough analysis of what went wrong, transforming incidents from purely negative events into valuable learning opportunities.
Well-structured runbooks enable consistent, rapid response to known issue patterns by externalizing operational knowledge and making it accessible to all team members regardless of experience level.
Effective communication practices keep stakeholders appropriately informed throughout incident lifecycle, managing expectations and maintaining confidence even during extended disruptions.
Continuous improvement through learning turns incident response from a purely reactive activity into a driver of organizational capability development, with each incident contributing to stronger systems, better processes, and more skilled teams.
Summary
Incident response and troubleshooting represent core competencies for infrastructure and platform management teams, directly impacting service reliability, user experience, and business outcomes. Effective incident management combines technical skills with organizational processes, communication practices, and cultural approaches that together enable rapid detection, systematic diagnosis, and efficient resolution of infrastructure issues.
The incident lifecycle provides a framework for understanding how incidents flow from initial detection through triage, active response, recovery, and ultimately learning. Each phase requires specific activities and capabilities, with successful incident management depending on strong execution across all phases rather than excellence in just one area. Organizations that invest in robust incident management processes—including clear role definitions, systematic troubleshooting methodologies, and comprehensive runbooks—consistently outperform those that rely on individual heroics and ad-hoc response approaches.
Root cause analysis and blameless postmortem culture transform incidents from purely reactive events into opportunities for organizational learning and systematic improvement. By distinguishing between proximate causes that immediately triggered incidents and root causes that created conditions where incidents could occur, teams identify improvements that prevent entire categories of similar issues rather than just addressing individual symptoms. Blameless culture ensures that learning can occur without the fear and information hiding that plague organizations where mistakes trigger blame and punishment.
The practices described in this chapter—from structured troubleshooting decision trees to comprehensive postmortem documentation to well-crafted runbooks—provide concrete tools that teams can immediately apply to improve their incident response capability. Yet these tactical practices are most effective when implemented within a broader organizational commitment to operational excellence, continuous learning, and building resilient systems that gracefully handle failures when they inevitably occur.