Chapter 11: Monitoring and Observability
Learning Objectives
After completing this chapter, you will be able to:
- Implement the three pillars of observability
- Design effective monitoring strategies
- Configure meaningful alerting
- Create operational dashboards
- Implement AIOps practices
- Build comprehensive observability pipelines
Introduction
The evolution from traditional monitoring to modern observability represents one of the most significant shifts in how organizations understand and manage their infrastructure. Traditional monitoring operated on a simple premise: define what can go wrong, create checks for those specific scenarios, and alert when thresholds are crossed. This approach worked adequately when systems were relatively simple, composed of monolithic applications running on physical servers with well-understood failure modes.
Modern infrastructure has fundamentally changed this equation. Distributed systems span multiple cloud providers, incorporate dozens of microservices, scale dynamically in response to load, and exhibit emergent behaviors that cannot be predicted in advance. In these complex environments, traditional monitoring falls short because it can only answer questions you know to ask. If your monitoring checks whether CPU exceeds 80%, it will alert on that condition—but it cannot tell you why user requests from a specific geographic region are experiencing intermittent timeouts at 3 AM on Tuesdays.
Observability addresses this gap by instrumenting systems to emit rich telemetry data that enables exploration of unknown questions. While monitoring tells you that something is wrong, observability helps you understand why it is wrong and how the system arrived at that state. This distinction is crucial: monitoring is about known unknowns (things you know can go wrong and have prepared for), while observability is about unknown unknowns (novel failure modes you have not encountered before).
The journey from monitoring to observability requires a fundamental shift in thinking. Instead of trying to predict every possible failure mode and create checks for each one, organizations instrument their systems to provide comprehensive visibility into their internal state and behavior. This telemetry data—metrics, logs, and traces—becomes the raw material for understanding system behavior, diagnosing problems, and making informed decisions about capacity, performance, and reliability.
This chapter explores the principles, tools, and practices for building comprehensive observability into your infrastructure. We will examine the three pillars that form the foundation of observability, explore strategies for effective monitoring, design meaningful alerting systems, create actionable dashboards, and investigate how artificial intelligence is augmenting human operators through AIOps capabilities. The goal is not merely to collect data but to transform that data into actionable insights that improve system reliability, performance, and the experience of both operators and end users.
The Three Pillars of Observability
The concept of three pillars—metrics, logs, and traces—has become the standard framework for thinking about observability. While the metaphor is imperfect (real observability requires more than just these three elements), it provides a useful mental model for understanding the different types of telemetry data and how they complement each other. Each pillar offers a different perspective on system behavior, and the real power of observability emerges when you can correlate signals across all three.
Understanding Metrics: The Quantitative Foundation
Metrics represent the quantitative heartbeat of your infrastructure. They are numerical measurements collected over time intervals, providing a statistical view of system behavior. Unlike logs that capture individual events or traces that follow specific requests, metrics aggregate behavior into summary statistics that can be efficiently stored, queried, and alerted upon. A single metric might represent millions of individual events, compressed into a time series that shows trends, patterns, and anomalies.
The power of metrics lies in their efficiency and queryability. When you need to understand whether your application is healthy, you do not want to sift through millions of log entries—you want to see a graph showing request rate, error rate, and response time over the past hour. Metrics provide this high-level view while consuming relatively little storage compared to raw event data. A year’s worth of metrics for an entire infrastructure might consume gigabytes of storage, while a single day’s logs could consume terabytes.
Modern metric systems distinguish between several fundamental types, each suited to different measurement scenarios. Counters are monotonically increasing values that track cumulative totals—the total number of requests served, the total number of errors encountered, or the total bytes transmitted. You typically work with counters using rate functions that calculate how quickly the counter is increasing, converting cumulative totals into rates like requests per second or errors per minute.
Gauges represent point-in-time measurements that can increase or decrease arbitrarily. CPU utilization, memory usage, queue depth, and connection counts are all gauges—they fluctuate based on current system state rather than accumulating over time. When you query a gauge, you see its current value or how that value has changed over a time window, making gauges ideal for tracking resource utilization and capacity.
Histograms and summaries provide statistical distributions rather than single values. When measuring request duration, you do not want just the average response time—averages hide critical information about outliers and distribution shape. A histogram captures the distribution of values across predefined buckets, enabling you to calculate percentiles like P50 (median), P95, and P99 on the server side. These percentiles are crucial for understanding user experience: while P50 tells you about typical performance, P99 tells you about the worst experiences your users encounter, and eliminating those worst-case scenarios often has the highest impact on perceived system quality.
# Prometheus recording rules for infrastructure
groups:
- name: infrastructure_metrics
rules:
# CPU utilization by instance
- record: instance:node_cpu_utilization:ratio
expr: |
1 - avg without(cpu, mode) (
rate(node_cpu_seconds_total{mode="idle"}[5m])
)
# Memory utilization by instance
- record: instance:node_memory_utilization:ratio
expr: |
1 - (
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes
)
# Disk I/O utilization
- record: instance:node_disk_io_utilization:ratio
expr: |
rate(node_disk_io_time_seconds_total[5m])
# Network receive/transmit rates
- record: instance:node_network_receive_bytes:rate5m
expr: |
rate(node_network_receive_bytes_total{device!="lo"}[5m])
- record: instance:node_network_transmit_bytes:rate5m
expr: |
rate(node_network_transmit_bytes_total{device!="lo"}[5m])
The recording rules shown above demonstrate a best practice in metric systems: pre-computing expensive queries and storing the results as new metrics. Instead of calculating CPU utilization every time someone views a dashboard, you compute it once per evaluation interval and store the result. This approach reduces query load on your metric system, speeds up dashboard rendering, and ensures consistent calculations across all dashboards and alerts.
Logs: The Detailed Event Chronicle
While metrics provide the quantitative overview, logs deliver the qualitative details. Every log entry represents a discrete event that occurred within your system—an API request was received, a database query executed, an error occurred, or a user logged in. Logs are the chronological story of what your system is doing, told through timestamped events with varying levels of detail and context.
The value of logs lies in their richness and specificity. When metrics indicate that error rates have increased, logs tell you exactly which errors occurred, for which users, in what context, and with what error messages. When investigating a production incident, operators often start with high-level metrics to identify when and where problems began, then dive into logs to understand the specific sequence of events that led to the issue.
Modern logging has evolved significantly from traditional text-based logs toward structured logging, where each log entry is a structured data object rather than a free-form text string. This evolution has profound implications for how logs can be searched, filtered, and analyzed. A structured log entry is typically represented as JSON, containing not just a message but also typed fields for all relevant context—user IDs, request IDs, service names, error types, and any other metadata needed to understand the event.
{
"timestamp": "2024-01-15T10:30:45.123Z",
"level": "ERROR",
"service": "payment-service",
"trace_id": "abc123def456",
"span_id": "xyz789",
"message": "Payment processing failed",
"error": {
"type": "PaymentGatewayError",
"message": "Connection timeout",
"stack": "..."
},
"context": {
"user_id": "user-12345",
"order_id": "order-67890",
"amount": 99.99,
"currency": "USD",
"gateway": "stripe"
},
"metadata": {
"environment": "production",
"region": "us-east-1",
"host": "payment-service-abc123",
"version": "2.3.1"
}
}
This structured approach transforms logs from write-only data (easy to write, hard to query) into queryable datasets. You can search for all errors from a specific user, all timeouts from a particular payment gateway, or all failures in a specific region. The trace_id field links this log entry to distributed traces, enabling you to follow this error back through the entire request chain to understand the full context of the failure.
Log aggregation architecture has become increasingly sophisticated to handle the volume and velocity of modern log data. A busy application might generate millions of log entries per minute, and these logs must be collected from ephemeral containers, parsed to extract structured fields, enriched with additional context, routed to appropriate storage systems, and indexed for fast searching—all while handling backpressure gracefully when downstream systems slow down.
The pipeline typically begins with log shippers running on each host or as sidecars alongside application containers. These shippers—tools like Fluentd, Filebeat, or Vector—read log files or consume logs from container runtimes, apply parsing rules to extract structure from text logs, add enrichment data like pod labels or cloud provider metadata, and forward the enriched logs to the next stage. For high-volume environments, a message queue like Kafka sits between shippers and storage, providing buffering and backpressure handling to prevent log loss during traffic spikes or storage system maintenance.
Log storage systems like Elasticsearch or Grafana Loki index the log data for fast querying. Elasticsearch provides full-text search across all log fields, enabling complex queries that combine text search, field filters, and aggregations. Loki takes a different approach, indexing only metadata labels while storing log content as compressed chunks, trading some query flexibility for significantly lower storage costs and simpler operations.
The choice of log retention periods reflects a balance between investigative needs and storage costs. System logs capturing operating system events typically have shorter retention (30 days), as most issues are detected and investigated quickly. Application logs with business context often warrant longer retention (90 days) to support debugging of intermittent issues and analyzing user behavior patterns. Audit logs recording security-relevant events must be retained for compliance periods, often a year or longer, and protected against tampering to maintain their evidentiary value.
Traces: The Request Journey Map
Distributed tracing completes the observability picture by showing how requests flow through complex, distributed systems. In a microservices architecture, a single user request might trigger calls to dozens of services, each performing its own work and potentially calling other services. Understanding the full path of a request, identifying where time is spent, and diagnosing where failures occur requires following the request across all these service boundaries—this is what distributed tracing provides.
A trace represents the complete journey of a request through your system. It consists of spans, where each span represents a single operation—a function call, a database query, an HTTP request to another service, or a message published to a queue. Spans are organized hierarchically: the root span represents the initial request, and child spans represent operations performed to fulfill that request. Each child span might have its own children, creating a tree structure that mirrors the call graph of your distributed system.
The power of tracing emerges when investigating latency issues. Metrics might tell you that P99 latency has increased from 200ms to 800ms, but they cannot tell you why. Traces let you select slow requests, examine their span trees, and identify exactly where time is being spent. You might discover that 90% of the added latency comes from a single database query that started performing poorly after a schema change, or that requests are spending 500ms in a retry loop because an upstream service is timing out intermittently.
+-----------------------------------------------------------------------+
| DISTRIBUTED TRACE EXAMPLE |
+-----------------------------------------------------------------------+
| |
| Trace ID: abc123def456 |
| |
| [Frontend] ───────────────────────────────────────────────────────── |
| │ Span: web-request (120ms) |
| │ |
| └─► [API Gateway] ────────────────────────────────────────────────── |
| │ Span: api-routing (5ms) |
| │ |
| └─► [Auth Service] ───────────────────────────────────────────── |
| │ Span: authenticate (15ms) |
| │ |
| └─► [Redis] ─────────────────────────────────────────────── │ |
| │ Span: cache-lookup (2ms) │ |
| └─────────────────────────────────────────────────────── |
| |
| └─► [Order Service] ──────────────────────────────────────────── |
| │ Span: process-order (80ms) |
| │ |
| ├─► [Inventory Service] ─────────────────────────────────── │ |
| │ │ Span: check-inventory (20ms) │ |
| │ │ │ |
| │ └─► [PostgreSQL] ────────────────────────────────────── │ |
| │ │ Span: db-query (10ms) │ |
| │ └─────────────────────────────────────────────────── |
| │ |
| └─► [Payment Service] ────────────────────────────────────── |
| │ Span: process-payment (45ms) |
| │ |
| └─► [Stripe API] ─────────────────────────────────────── |
| │ Span: external-call (40ms) |
| └─────────────────────────────────────────────────── |
| |
+-----------------------------------------------------------------------+
This trace visualization reveals several insights that would be invisible in metrics alone. The total request duration is 120ms, with most time spent in order processing (80ms). Within order processing, the payment service call (45ms) dominates, and most of that time is spent calling the external Stripe API (40ms). If you need to improve latency, optimizing the inventory service (20ms) would have less impact than optimizing the payment flow—perhaps by implementing payment method tokenization to reduce external API calls.
Implementing distributed tracing requires propagating trace context across service boundaries. When Service A calls Service B, it must pass along the trace ID and span ID so Service B can create child spans that link back to the parent. This propagation traditionally required each service to understand a specific tracing format, but the OpenTelemetry project has standardized this process, providing libraries for all major programming languages that handle context propagation automatically.
# OpenTelemetry Collector configuration
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 4096
spike_limit_mib: 512
resource:
attributes:
- key: environment
value: production
action: upsert
exporters:
jaeger:
endpoint: jaeger:14250
tls:
insecure: true
prometheus:
endpoint: 0.0.0.0:8889
loki:
endpoint: http://loki:3100/loki/api/v1/push
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [jaeger]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loki]
The OpenTelemetry Collector configuration shown above demonstrates the modern approach to telemetry collection. Rather than having each service send data directly to multiple backend systems (Prometheus for metrics, Loki for logs, Jaeger for traces), services send all telemetry to a single collector using the OpenTelemetry Protocol (OTLP). The collector then handles routing to appropriate backends, providing a clean separation between instrumentation and storage.
This architecture offers several advantages. Services need to know only about the collector endpoint, not about every backend system. The collector can perform processing like batching, filtering, and enrichment, reducing load on backend systems and network overhead. If you need to add a new backend or change existing ones, you update the collector configuration rather than reconfiguring every service. The collector also provides buffering and retry logic, preventing telemetry loss during transient network issues or backend maintenance windows.
Correlating the Three Pillars
The true power of observability emerges not from collecting metrics, logs, and traces independently, but from correlating them to build a complete picture of system behavior. When investigating an incident, you typically start with metrics—they show you when the problem started and which components are affected. Metrics lead you to relevant logs, which provide detailed error messages and context. Logs contain trace IDs that let you examine specific request flows through distributed tracing. This iterative process of moving between pillars, each providing different perspectives and deeper context, is how modern incident response works.
Consider investigating a customer report that checkout is slow. You begin with application metrics showing that P99 latency for the checkout service has increased from 300ms to 2000ms over the past hour. You filter logs for the checkout service during this time window, discovering numerous timeout errors when calling the inventory service. The log entries contain trace IDs, so you examine traces for these failed requests, discovering that the inventory service is experiencing database query timeouts. Metrics for the database server show that disk I/O latency has spiked. System logs for the database server reveal that a backup job started an hour ago and is saturating disk bandwidth.
In this investigation, each pillar provided essential information that the others could not. Metrics identified the symptom and affected component. Logs provided error details and linked to traces. Traces showed the request flow and identified the bottleneck. More metrics identified the resource contention. System logs revealed the root cause. Without any one of these pillars, the investigation would have been significantly more difficult or even impossible.
Monitoring Strategy
Building an effective monitoring strategy requires thinking systematically about what you need to observe, why you need to observe it, and how the telemetry data will be used. Too often, organizations fall into the trap of monitoring everything because they can, resulting in overwhelming amounts of data with unclear value. An effective strategy focuses on metrics that inform decisions and enable action.
Understanding the Golden Signals
Site Reliability Engineering, as practiced at Google and documented in the SRE books, introduced the concept of Golden Signals—four key metrics that provide a comprehensive view of service health. These signals represent the minimum set of metrics needed to understand whether a service is operating correctly and serving users effectively. While additional metrics can provide deeper insights, the Golden Signals should always be measured and monitored for every user-facing service.
Latency measures how long it takes to service requests. Critically, you must distinguish between the latency of successful requests and the latency of failed requests. A failing request might be fast (immediately returning an error) or slow (timing out after 30 seconds), and these different failure modes require different responses. Measuring latency using percentiles rather than averages is essential—P50 tells you about typical performance, P95 shows that 95% of requests are faster than this value, and P99 reveals the experience of your worst-served users. For user-facing services, P99 latency often matters more than average latency, as even a small percentage of slow requests can significantly impact user satisfaction.
Traffic measures the demand on your system, quantified in requests per second, transactions per minute, or whatever unit makes sense for your service. Traffic patterns reveal usage trends, help identify capacity needs, and provide context for other metrics. A spike in latency during a traffic surge might be expected and acceptable, while the same latency spike during normal traffic suggests a problem. Traffic metrics also enable capacity planning—by understanding traffic growth trends, you can provision resources proactively rather than reacting to capacity exhaustion.
Errors track the rate of failed requests, whether explicitly failed (HTTP 500 errors, exceptions thrown) or implicitly failed (wrong content returned, policy violations, timeouts). The error rate is typically expressed as a percentage of total requests, making it easy to set thresholds like “alert if error rate exceeds 1%.” However, not all errors are equal—a 500 error indicating a service crash is more severe than a 404 error indicating a missing resource. Sophisticated error monitoring distinguishes between error types and weights them appropriately.
Saturation measures how “full” your service is, indicating how close you are to maximum capacity. For compute resources, saturation might be CPU and memory utilization. For databases, it could be connection pool usage or disk I/O. For message queues, it is queue depth. Saturation metrics are predictive—they warn you that capacity limits are approaching before you hit them, enabling proactive action. When saturation approaches 100%, performance typically degrades non-linearly, so the goal is to operate comfortably below saturation limits with enough headroom for traffic spikes.
# Latency - P99 response time
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{job="api"}[5m]))
by (le)
)
# Traffic - Requests per second
sum(rate(http_requests_total{job="api"}[5m]))
# Errors - Error rate
sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api"}[5m]))
# Saturation - CPU utilization
avg(1 - rate(node_cpu_seconds_total{mode="idle"}[5m]))
These PromQL queries demonstrate how to calculate Golden Signals using Prometheus metrics. The latency query uses histogram_quantile to extract the 99th percentile from histogram buckets, calculated over a 5-minute window using rate(). The traffic query simply sums the rate of all requests. The error rate divides the rate of 5xx errors by the total rate of all requests, producing a ratio that can be compared against SLOs. The saturation query calculates CPU utilization by subtracting idle time from 100%.
Layered Monitoring Approach
Effective monitoring covers multiple layers of the technology stack, from infrastructure through applications to business metrics. Each layer provides different insights and serves different audiences. Infrastructure metrics inform capacity decisions and hardware failures. Platform metrics help operators understand cluster health and resource distribution. Application metrics guide developers in optimizing code and diagnosing bugs. Business metrics show executives whether the system is achieving its purpose of serving users and generating value.
Infrastructure monitoring focuses on the physical or virtual resources that run your workloads—servers, networks, storage systems, and virtualization platforms. Key metrics include CPU utilization, memory usage, disk I/O throughput and latency, network bandwidth and errors, and system-level error rates. These metrics often follow standard patterns: utilization should remain below safe thresholds (typically 70-85%), error rates should be zero or near-zero, and latency should remain consistently low. Infrastructure monitoring often catches issues before they impact applications, as resource exhaustion typically manifests first as degraded infrastructure metrics before causing application failures.
Platform monitoring covers orchestration systems like Kubernetes, database clusters, message brokers, and other middleware that sits between infrastructure and applications. For Kubernetes, this includes pod status, node capacity, deployment health, service mesh metrics, and control plane API latency. For databases, it covers connection pool usage, query performance, replication lag, and transaction rates. Platform metrics help operators understand whether the platform itself is healthy and whether it has sufficient capacity for current workloads.
Application monitoring instruments your code to measure business-relevant metrics—user requests, API calls, background jobs, cache hit rates, and feature-specific metrics. Application metrics are where observability becomes specific to your organization and use cases. While infrastructure and platform metrics are relatively universal (all Kubernetes clusters expose similar metrics), application metrics reflect your unique business logic and user journeys. These metrics should be designed to answer questions about user experience: are users able to complete critical workflows? Are features performing as expected? Where are users encountering errors?
Business metrics connect technical operations to business outcomes. Revenue per transaction, conversion rates, user engagement metrics, and transaction success rates bridge the gap between operators who manage systems and executives who manage business strategy. When these metrics degrade, the organization knows immediately that technical issues are impacting business results, mobilizing appropriate response. Many organizations create dashboards showing both technical metrics and business metrics side-by-side, making the connection between system health and business impact explicit.
Alerting Design
Alerting transforms passive monitoring into active incident response by notifying operators when problems require human attention. However, alerting is a double-edged sword—too few alerts and critical issues go unnoticed until customers report them; too many alerts and operators become desensitized, ignoring or dismissing all alerts including the critical ones. Effective alerting requires careful design focused on actionability, appropriate severity levels, clear escalation paths, and continuous tuning.
The Philosophy of Actionable Alerts
The fundamental principle of good alerting is simple: every alert should require human action. If an alert does not require action—either to investigate, mitigate, or fix a problem—it should not be an alert. This principle seems obvious but is violated constantly in practice. Organizations create alerts for informational events (“deployment succeeded”), for transient conditions that self-resolve (“CPU spiked to 85% for 30 seconds”), or for conditions that someone should know about eventually but not immediately (“disk will be full in 90 days”).
These non-actionable alerts create alert fatigue, a condition where operators become conditioned to ignore or dismiss alerts because most alerts do not represent real problems requiring immediate action. Alert fatigue is insidious because it affects even critical alerts—when your pager goes off five times per night for non-issues, you start to assume the sixth page is also a non-issue. The solution is ruthless curation: delete, reclassify, or tune alerts until only actionable ones remain.
Actionability requires that alerts provide enough context for operators to begin investigating immediately. An alert saying “service unhealthy” is not actionable—where is the service? What specifically is unhealthy? What are the symptoms? An actionable alert includes the affected component, the specific symptom or metric that triggered the alert, current values and thresholds, relevant diagnostic information, links to dashboards showing the service’s current state, and a link to the runbook documenting investigation and remediation steps.
Every alert should have an associated runbook—a document that explains what the alert means, what could cause it, how to investigate, and how to remediate common root causes. The first time an alert fires, the person investigating writes down their investigation process, their findings, and how they resolved the issue. This becomes the first version of the runbook. Each subsequent occurrence refines the runbook based on new failure modes and investigation techniques. Over time, the runbook becomes a comprehensive guide that enables any operator to handle the alert effectively, even if they have never seen it before.
Severity Levels and Response Times
Alert severity levels map alerts to appropriate response procedures and response times. Different organizations use different severity schemes, but a typical approach has four levels: Critical alerts (P1) indicate complete service outages, data loss risks, or security breaches requiring immediate response regardless of time of day. High priority alerts (P2) indicate significant service degradation that will likely escalate to critical if not addressed soon, requiring response within an hour. Warning alerts (P3) indicate concerning trends or minor issues that need attention during business hours. Informational notices (P4) document notable events but require no immediate response.
The critical distinction between severity levels is response time and interrupt urgency. Critical alerts page on-call engineers immediately via phone and SMS, waking them at night if necessary, because waiting until morning could result in catastrophic impact. High priority alerts go to persistent channels like Slack or email, where they will be seen within an hour but do not interrupt sleep. Warning alerts go to dashboards and ticket systems where they will be reviewed during business hours. Informational notices are recorded but not actively pushed to operators.
Misclassification of severity causes major problems in both directions. If genuinely critical alerts are classified as warnings, they may not be seen until significant damage has occurred. Conversely, if minor issues are classified as critical, operators experience alert fatigue and may become desensitized to actual critical alerts. Organizations must regularly review alert severity classification and adjust based on actual impact—if an alert has never represented an actual critical issue, downgrade it; if a warning alert has escalated to production-impacting incidents multiple times, upgrade it.
groups:
- name: infrastructure_alerts
rules:
# High CPU utilization
- alert: HighCPUUtilization
expr: |
instance:node_cpu_utilization:ratio > 0.85
for: 10m
labels:
severity: warning
team: infrastructure
annotations:
summary: "High CPU utilization on "
description: |
CPU utilization is
for more than 10 minutes.
runbook_url: "https://wiki.example.com/runbooks/high-cpu"
dashboard_url: "https://grafana.example.com/d/nodes?var-instance="
# Critical CPU utilization
- alert: CriticalCPUUtilization
expr: |
instance:node_cpu_utilization:ratio > 0.95
for: 5m
labels:
severity: critical
team: infrastructure
annotations:
summary: "Critical CPU utilization on "
description: |
CPU utilization is critically high at .
Immediate action required.
runbook_url: "https://wiki.example.com/runbooks/critical-cpu"
# Disk space low
- alert: DiskSpaceLow
expr: |
(node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.15
for: 15m
labels:
severity: warning
annotations:
summary: "Low disk space on "
description: |
Disk has only
free space.
# Memory pressure
- alert: HighMemoryUtilization
expr: |
instance:node_memory_utilization:ratio > 0.90
for: 10m
labels:
severity: warning
annotations:
summary: "High memory utilization on "
# Error rate spike
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: "Error rate is "
These Prometheus alerting rules demonstrate several best practices. Each alert has a clear name describing what it detects. The for clause ensures that transient spikes do not trigger alerts—CPU must remain above threshold for 10 minutes before alerting. Labels categorize alerts by severity and owning team, enabling routing to appropriate responders. Annotations provide rich context including current values, links to relevant dashboards, and links to runbooks. The HighCPUUtilization and CriticalCPUUtilization alerts show a common pattern of having multiple severity levels for the same condition, with different thresholds and response procedures.
Alert Routing and Escalation
Once an alert fires, it must reach the appropriate responders through appropriate channels. Alertmanager, the alerting component of the Prometheus ecosystem, provides sophisticated routing capabilities that map alerts to notification channels based on alert labels. A critical alert from the payment service goes to the payments team via PagerDuty and Slack. A warning from infrastructure goes to the infrastructure team’s Slack channel. An alert from a development environment goes to a dashboard but nowhere else.
Escalation policies ensure that critical alerts receive response even if the primary on-call engineer is unavailable. A typical escalation policy first attempts to contact the primary on-call engineer via phone and SMS. If they do not acknowledge the page within 5 minutes, it escalates to the secondary on-call engineer. After another 5 minutes without acknowledgment, it escalates to the engineering manager. This escalation prevents alerts from being missed due to temporary unavailability while avoiding unnecessary interruptions when the primary responds promptly.
Inhibition rules prevent alert storms by suppressing dependent alerts when root cause alerts are active. If a server is completely down, dozens of alerts might fire—services on that server are unreachable, health checks are failing, metrics are missing. Rather than paging for each of these symptoms, inhibition rules suppress the symptom alerts when the root cause alert (server down) is active. This dramatically reduces noise during incidents, allowing responders to focus on the actual problem rather than triaging dozens of related alerts.
# Alertmanager configuration
global:
resolve_timeout: 5m
slack_api_url: 'https://hooks.slack.com/services/xxx'
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'default'
routes:
- match:
severity: critical
receiver: 'pagerduty-critical'
continue: true
- match:
severity: critical
receiver: 'slack-critical'
- match:
severity: warning
receiver: 'slack-warnings'
- match:
team: infrastructure
receiver: 'infrastructure-team'
receivers:
- name: 'default'
slack_configs:
- channel: '#alerts'
- name: 'pagerduty-critical'
pagerduty_configs:
- service_key: '<pagerduty-key>'
severity: critical
- name: 'slack-critical'
slack_configs:
- channel: '#alerts-critical'
title: ': '
text: ''
- name: 'slack-warnings'
slack_configs:
- channel: '#alerts-warnings'
- name: 'infrastructure-team'
slack_configs:
- channel: '#infrastructure-alerts'
email_configs:
- to: 'infrastructure@example.com'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'instance']
This Alertmanager configuration demonstrates sophisticated routing. The global route groups alerts by name, cluster, and service, waiting 30 seconds to batch related alerts together before sending notifications. Critical alerts go to both PagerDuty (via continue: true allowing processing to continue to subsequent routes) and Slack. Warning alerts go only to Slack. Alerts tagged with team=infrastructure additionally go to team-specific channels and email. The inhibit rule prevents warning alerts from being sent when a critical alert for the same alertname and instance is already active.
Dashboards
Dashboards transform raw telemetry data into visual insights that enable rapid understanding of system state, identification of anomalies, and communication of status to stakeholders. An effective dashboard tells a story at a glance—is the system healthy or degraded? Where are problems occurring? What is the trend over time? However, creating truly effective dashboards requires understanding your audience, applying information design principles, and resisting the temptation to include everything.
Dashboard Design Principles
The first principle of effective dashboard design is purpose—every dashboard should have a clear, single purpose. An operational health dashboard shows real-time service status and alerts, consumed by on-call engineers responding to incidents. An executive dashboard shows SLO compliance and business metrics, consumed by leadership making strategic decisions. A debugging dashboard shows detailed metrics for a specific service, consumed by engineers investigating performance issues. A single dashboard cannot serve all these purposes effectively; trying to do so results in cluttered, confusing displays that serve no audience well.
The second principle is hierarchy—the most important information should be the most prominent. When an engineer opens an operational dashboard during an incident, they need to see immediately whether there is an active problem, where it is occurring, and how severe it is. This critical information belongs at the top in large, easy-to-read formats. Detailed metrics, historical trends, and supplementary information belong below, available when needed but not competing for attention with critical signals.
The third principle is context—metrics are meaningful only in relation to baselines, thresholds, and comparisons. A graph showing 45ms latency provides little information by itself. Is 45ms good or bad? Is it increasing or stable? Add a horizontal line at 100ms (your SLO threshold) and color the area red when latency exceeds it, and the metric becomes immediately interpretable. Add a lighter line showing yesterday’s latency for comparison, and you can see whether current behavior is anomalous.
The fourth principle is actionability—dashboards should enable action, not just observation. Include links to runbooks, to more detailed debugging dashboards, to log queries filtered to relevant time ranges, and to systems that enable taking action like deployment tools or scaling controls. When an engineer identifies a problem on a dashboard, they should be able to click through to investigate details and potentially remediate, all without leaving the dashboard environment.
The fifth principle is simplicity—resist the urge to include every metric. A dashboard with 50 graphs is not 10 times better than one with 5 graphs; it is 10 times harder to understand. Each panel should justify its inclusion by answering an important question that the dashboard’s audience needs answered. If a metric is interesting but not essential, it belongs on a supplementary dashboard, not the main operational view.
+-----------------------------------------------------------------------+
| OPERATIONS DASHBOARD LAYOUT |
+-----------------------------------------------------------------------+
| |
| ROW 1: KEY METRICS (at-a-glance health) |
| +----------+ +----------+ +----------+ +----------+ +----------+ |
| | Uptime | | Error | | Latency | | Traffic | | Open | |
| | 99.95% | | 0.1% | | 45ms | | 5.2k/s | | Incidents| |
| +----------+ +----------+ +----------+ +----------+ +----------+ |
| |
| ROW 2: TRENDS (time-series graphs) |
| +----------------------------------+ +------------------------------+ |
| | | | | |
| | Request Rate (24h) | | Error Rate (24h) | |
| | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ | | ___________________________ | |
| | | | | |
| +----------------------------------+ +------------------------------+ |
| |
| ROW 3: RESOURCE UTILIZATION |
| +------------------+ +------------------+ +------------------+ |
| | CPU by Host | | Memory by Host | | Disk by Mount | |
| | ============ | | ============ | | ============ | |
| | host-1: 45% | | host-1: 62% | | /data: 72% | |
| | host-2: 38% | | host-2: 58% | | /var: 45% | |
| +------------------+ +------------------+ +------------------+ |
| |
| ROW 4: TOP ISSUES |
| +---------------------------------------------------------------+ |
| | Recent Alerts | Active Incidents | |
| | - High CPU on prod-web-03 | - INC-123: API degradation | |
| | - Disk space warning prod-db | - INC-124: Payment timeout | |
| +---------------------------------------------------------------+ |
| |
+-----------------------------------------------------------------------+
This dashboard layout demonstrates the hierarchy principle in action. Row 1 contains single-value stat panels showing the most critical metrics—uptime, error rate, current latency, traffic rate, and open incident count. These answer the first question any operator has: “Is everything okay?” Row 2 shows time-series trends for request rate and error rate, enabling operators to see whether current values are normal or anomalous. Row 3 shows resource utilization broken down by host, identifying capacity constraints. Row 4 surfaces recent alerts and active incidents, providing quick access to current issues requiring attention.
Dashboards for Different Audiences
Executive dashboards focus on business impact and SLO compliance rather than technical details. They answer questions like “Are we meeting our reliability commitments?” and “How is system performance affecting business metrics?” An executive dashboard might show SLO compliance over time, broken down by service; incident frequency and mean time to resolution trends; infrastructure costs and cost efficiency metrics; and key business metrics like transaction success rate and user engagement. These dashboards typically update daily or weekly rather than in real-time, as executives make strategic decisions on longer time horizons than operators make tactical decisions.
Operational dashboards serve engineers managing production systems in real-time. They prioritize current system state, active alerts, and recent trends. An operational dashboard opens on a large screen in the NOC or automatically when an alert fires, providing immediate context about what is happening. These dashboards update in real-time, often showing metrics aggregated over short windows like 5 minutes to quickly reveal emerging issues. They include navigation to detailed service dashboards and debugging tools, enabling smooth transition from “something is wrong” to “I understand what is wrong” to “I am fixing what is wrong.”
Service-specific dashboards provide detailed visibility into individual services for the teams that own them. While operational dashboards show high-level health across all services, service dashboards show everything about one service—all its metrics, its dependencies, its resource utilization, and service-specific metrics that only make sense in the context of that particular service. Service dashboards are where product teams spend time optimizing performance, capacity planning, and debugging issues reported by users.
AIOps and Machine Learning
Artificial intelligence for IT operations, commonly known as AIOps, represents the application of machine learning and artificial intelligence to the operational problems of managing complex infrastructure. As systems grow in complexity, the volume of telemetry data overwhelms human capacity to monitor effectively. A large infrastructure might generate millions of metric time series, billions of log entries per day, and thousands of alerts per week. AIOps techniques help operators extract signal from this noise, identify patterns that humans might miss, and automate responses to routine issues.
Anomaly Detection: Finding the Unusual
Anomaly detection applies statistical and machine learning techniques to identify unusual patterns in metrics, logs, and traces. Traditional threshold-based alerting requires knowing in advance what values indicate problems—CPU above 80%, error rate above 1%, latency above 200ms. This approach works well for known failure modes but cannot detect novel issues that do not cross predefined thresholds. Perhaps latency is within normal bounds at 150ms, but it is unusual for this time of day when it is typically 50ms. Perhaps error rate is 0.5%, below your threshold, but it was 0% all of last month, so 0.5% represents a significant degradation.
Anomaly detection algorithms learn the normal behavior of each metric and identify deviations from that learned baseline. Simple approaches use statistical methods like standard deviation—if a metric is more than 3 standard deviations from its mean, flag it as anomalous. More sophisticated approaches use time-series forecasting to predict expected values based on historical patterns, then alert when actual values deviate significantly from predictions. Machine learning approaches can capture complex patterns like seasonality (traffic is higher during business hours), day-of-week effects (traffic is lower on weekends), and trends (traffic grows 5% month over month).
The challenge with anomaly detection is tuning sensitivity to minimize false positives while catching real anomalies. If the system flags 100 anomalies per day but only 2 represent real issues, operators will ignore the anomaly alerts just as they ignore poorly-tuned threshold alerts. Effective anomaly detection requires iterative refinement, incorporating feedback about which detected anomalies were real issues and which were false positives, gradually improving the model’s accuracy.
Event Correlation: Connecting Related Signals
In a complex distributed system, a single root cause often manifests as dozens or hundreds of symptoms visible in different metrics and logs. A failed database server might cause elevated error rates in multiple application services, increased retry rates in message queues, elevated latency for database-dependent APIs, and missing metrics from the database exporter. Traditional alerting fires separate alerts for each symptom, creating alert storms that overwhelm operators and obscure the underlying issue.
Event correlation groups related events together, identifying that all these alerts likely stem from a single root cause. The simplest correlation is temporal—events occurring at the same time are probably related. More sophisticated correlation considers the topology of your infrastructure, grouping events from components that depend on each other. If the database fails, the correlation engine knows that applications depending on that database will also show symptoms, so it groups all those alerts together and identifies the database failure as the probable root cause.
Advanced correlation systems use machine learning to discover correlation patterns from historical data. They observe that whenever metric X spikes, metrics Y and Z also spike a few minutes later, even if there is no explicit dependency configured between these components. By learning these empirical relationships, correlation systems can identify novel relationships that operators might not have explicitly documented.
Predictive Analytics: Seeing the Future
Predictive analytics applies forecasting techniques to anticipate future problems before they occur. The most straightforward application is capacity forecasting—by analyzing trends in disk usage, you can predict when disks will fill and proactively allocate additional storage before running out of space. Similarly, forecasting traffic growth enables proactive capacity planning, ensuring you add servers before hitting capacity limits rather than scrambling to scale during a traffic spike.
More advanced predictive systems forecast failures by identifying leading indicators. Perhaps specific patterns in system metrics predict hardware failures hours before they occur—increasing disk errors, elevated memory error rates, or temperature anomalies. By detecting these patterns, you can proactively migrate workloads off failing hardware before it crashes, preventing service disruptions rather than merely responding to them quickly.
The challenge with predictive analytics is achieving sufficient accuracy that predicted issues are taken seriously. If predictions are frequently wrong, operators stop trusting them and ignore warnings about real impending problems. Building trust requires starting with high-confidence predictions (disk will be full in 7 days based on current usage trends) before moving to more speculative predictions (this server will likely fail in the next 48 hours based on anomalous metric patterns).
Automated Remediation: Closing the Loop
The ultimate goal of AIOps is not just to detect and diagnose problems faster but to automatically remediate them without human intervention. The simplest forms of auto-remediation have been common for years—automatically restarting crashed processes, automatically scaling applications in response to traffic, automatically failing over to standby systems when primaries fail. AIOps extends these capabilities by applying intelligence to determine which remediation actions are appropriate for novel situations.
An AIOps system might observe that a specific service becomes unhealthy after deployment, automatically roll back the deployment, and page the engineering team with a report of what went wrong. It might detect that a microservice is experiencing elevated latency due to resource constraints, automatically increase its resource allocation, and file a ticket for capacity planning review. It might identify a security anomaly like unusual API access patterns, automatically block the suspicious traffic, and alert the security team for investigation.
The key to successful auto-remediation is confidence and safety. Automated actions should only be taken when the system is highly confident in its diagnosis and the remediation action is safe. Rolling back a deployment that is causing errors is relatively safe—worst case, you keep running the previous version a bit longer. Automatically deleting data to free up disk space is risky—you might delete something important. Building trust in auto-remediation requires starting with safe, reversible actions and gradually expanding the set of automated remediations as the system proves its reliability.
Review Questions
- How do metrics, logs, and traces complement each other in observability, and what types of questions is each pillar best suited to answer?
- What are the four Golden Signals, and why do Site Reliability Engineers consider them sufficient for understanding service health?
- How should alert severity levels map to response procedures, and why is it critical to avoid misclassifying alert severity?
- What design principles make dashboards effective for their intended audiences, and why is trying to create one dashboard for all audiences problematic?
- How can AIOps techniques like anomaly detection and event correlation reduce alert fatigue while improving incident detection and response?
Key Takeaways
- Observability transcends traditional monitoring by enabling exploration of unknown questions rather than just checking predefined conditions, making it essential for understanding complex distributed systems
- The three pillars of observability (metrics, logs, and traces) provide complementary perspectives, with the real power emerging from correlating signals across all three pillars
- Golden Signals (latency, traffic, errors, saturation) provide a focused set of metrics that comprehensively indicate service health without drowning operators in excessive data
- Effective alerting requires ruthless focus on actionability, ensuring every alert requires human action and provides sufficient context for immediate investigation
- Dashboards must be designed for specific audiences with clear hierarchy, appropriate detail levels, and actionable insights rather than attempting to display all available metrics
- AIOps augments human capabilities through anomaly detection, event correlation, predictive analytics, and automated remediation, handling the scale and complexity that overwhelms human operators
Summary
Monitoring and observability form the foundation for operating reliable, performant infrastructure in production. The journey from traditional threshold-based monitoring to modern observability reflects the evolution from simple, predictable systems to complex, distributed architectures with emergent behaviors. By instrumenting systems to emit comprehensive telemetry across the three pillars of metrics, logs, and traces, organizations gain the visibility needed to understand system behavior, diagnose problems rapidly, and make informed decisions about capacity, performance, and reliability. Effective alerting ensures that operators receive timely notification of issues requiring action without suffering alert fatigue from excessive noise. Well-designed dashboards present this telemetry data in forms appropriate for different audiences, from executives tracking business impact to engineers debugging production issues. AIOps techniques increasingly augment human operators by automatically detecting anomalies, correlating related events, predicting future issues, and even remediating problems without human intervention. The result is infrastructure that is not merely monitored but truly observable—where any question about system behavior can be answered through exploration of telemetry data, where problems are detected before they impact users, and where operators have the insights needed to continuously improve reliability and performance.