{"title":"SLO, SLI, SLA","description":"","content":"Your team is arguing about reliability. The product manager wants five-nines availability (99.999%). Engineering says that would mean almost no releases. The CEO read about SLAs in a contract and wants to know why you do not have one. Meanwhile, no one can agree on what \"good enough\" actually means.\n\nThis confusion is common. Teams chase arbitrary reliability targets without understanding the cost. They commit to SLAs without measuring if they can achieve them. They alert on thresholds that have no connection to user happiness.\n\nSLIs, SLOs, and SLAs provide a framework for defining, measuring, and committing to reliability. They answer fundamental questions: What should we measure? How reliable should we be? What do we promise to customers?\n\nIn this chapter, you will learn:\n\n- The difference between SLI, SLO, and SLA\n- How to define meaningful SLIs\n- Setting realistic SLOs\n- Error budgets and how to use them\n- SLA considerations for external commitments\n\nThis framework connects observability to business decisions. It turns abstract metrics into actionable reliability targets.\n\n---\n\n# The Three Concepts\n\nLet us start with clear definitions:\n\n\n```mermaid\nflowchart TB\n subgraph Definitions\n SLI[SLI
Service Level Indicator
What we measure]:::primary\n SLO[SLO
Service Level Objective
What we target]:::orange\n SLA[SLA
Service Level Agreement
What we promise]:::red\n end\n\n SLI --> |\"feeds into\"| SLO --> |\"informs\"| SLA\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n\n| Term | Definition | Example |\n|------|------------|---------|\n| **SLI** | A quantitative measure of service behavior | \"99.3% of requests completed in under 200ms\" |\n| **SLO** | Target value for an SLI | \"99.9% of requests should complete in under 200ms\" |\n| **SLA** | Contract with consequences for missing targets | \"If uptime falls below 99.9%, customer gets credits\" |\n\n\nThink of it this way:\n\n- **SLI** = What is the speedometer reading?\n- **SLO** = What speed should we maintain?\n- **SLA** = What happens if we drive too slow?\n\n---\n\n# Service Level Indicators (SLIs)\n\nAn SLI is a metric that reflects user experience. Not all metrics are SLIs. CPU usage is a metric. The percentage of requests that succeed is an SLI.\n\n### Characteristics of Good SLIs\n\nGood SLIs are:\n\n**1. User-centric** Measure what users care about, not internal system health.\n\n\n```shell\nGood SLI: Percentage of page loads under 3 seconds\nBad SLI: Average database query time\n```\n\n\n**2. Measurable** You must be able to collect the data reliably.\n\n\n```shell\nGood SLI: Request success rate (measured at load balancer)\nBad SLI: User satisfaction (hard to measure automatically)\n```\n\n\n**3. Specific** Define exactly what is measured and how.\n\n\n```shell\nGood SLI: \"Percentage of HTTP requests returning 2xx or 4xx within 500ms,\n measured at the API gateway, excluding health checks\"\nBad SLI: \"Response time\"\n```\n\n\n### Common SLI Categories\n\n\n```mermaid\nflowchart TB\n subgraph \"SLI Categories\"\n A[Availability
Did it work?]:::primary\n L[Latency
How fast was it?]:::orange\n T[Throughput
How much can it handle?]:::teal\n Q[Quality
How good was it?]:::purple\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n**Availability SLIs:**\n\n- Percentage of successful requests\n- Percentage of time service responds to health checks\n- Percentage of scheduled jobs that complete successfully\n\n**Latency SLIs:**\n\n- Percentage of requests under threshold (p50, p95, p99)\n- Median response time\n- Time to first byte\n\n**Throughput SLIs:**\n\n- Requests processed per second during peak\n- Messages processed within time window\n\n**Quality SLIs:**\n\n- Percentage of search results rated relevant\n- Percentage of recommendations clicked\n- Data freshness (percentage of data updated within threshold)\n\n### SLI Specification Template\n\nDocument each SLI precisely:\n\n\n```shell\nSLI Name: Request Latency\n─────────────────────────────────────\nDescription: Percentage of HTTP requests that complete within 500ms\nMeasurement: Response time measured at API gateway\nIncludes: All GET and POST requests to /api/* endpoints\nExcludes: Health checks, admin endpoints\nGood event: Response time < 500ms AND status code 2xx or 4xx\nValid event: Any request to included endpoints\nCalculation: (good events / valid events) * 100\n```\n\n\n---\n\n# Service Level Objectives (SLOs)\n\nAn SLO is a target value for an SLI. It defines \"good enough\" reliability.\n\n### Setting SLOs\n\n\n```mermaid\nflowchart LR\n subgraph Inputs\n U[User expectations]:::primary\n H[Historical performance]:::primary\n C[Cost of reliability]:::primary\n B[Business requirements]:::primary\n end\n\n D[Define SLO]:::orange\n\n U --> D\n H --> D\n C --> D\n B --> D\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nConsider these factors:\n\n**User expectations:** What do users consider acceptable? A consumer app needs low latency. A batch processing system can be slower.\n\n**Historical performance:** What has the system actually achieved? Setting an SLO below current performance is pointless. Setting it far above creates impossible targets.\n\n**Cost of reliability:** Each nine of availability costs more. Going from 99% to 99.9% requires different architecture than going from 99.9% to 99.99%.\n\n**Business requirements:** Does the business require specific guarantees for contracts or competitive reasons?\n\n### The Cost of Nines\n\n\n```shell\nAvailability Downtime/year Downtime/month Downtime/week\n───────────── ───────────── ────────────── ─────────────\n99% 3.65 days 7.2 hours 1.68 hours\n99.9% 8.76 hours 43.2 minutes 10.1 minutes\n99.95% 4.38 hours 21.6 minutes 5.0 minutes\n99.99% 52.56 minutes 4.32 minutes 1.0 minutes\n99.999% 5.26 minutes 25.9 seconds 6.0 seconds\n```\n\n\nEach additional nine is exponentially harder to achieve:\n\n\n| Target | What It Requires |\n|--------|------------------|\n| **99%** | Basic redundancy, manual failover |\n| **99.9%** | Automated failover, multi-zone deployment |\n| **99.99%** | Multi-region, sophisticated automation, limited releases |\n| **99.999%** | Massive investment, possibly multiple providers, near-zero releases |\n\n\n**Interview Insight:** When asked about reliability targets, discuss the trade-offs. Five nines sounds impressive but may be unnecessary and extremely expensive. Ask what the business actually needs.\n\n### SLO Examples\n\n\n| Service Type | SLI | SLO |\n|--------------|-----|-----|\n| **E-commerce checkout** | Successful transactions | 99.95% success rate |\n| **API service** | Request latency | 99% of requests < 200ms |\n| **Search engine** | Query latency | p99 latency < 500ms |\n| **Streaming video** | Playback starts | 99.9% start within 2s |\n| **Batch processing** | Job completion | 99% complete within window |\n\n\n### SLO Windows\n\nSLOs are measured over a time window:\n\n\n```shell\nRolling window: \"99.9% availability over the last 30 days\"\n- Continuously calculated\n- Smooths out short incidents\n- More commonly used\n\nCalendar window: \"99.9% availability each month\"\n- Resets at period boundaries\n- Easier to report on\n- Can create end-of-period risk\n```\n\n\n---\n\n# Error Budgets\n\nThe error budget is the inverse of your SLO. If your SLO is 99.9% availability, your error budget is 0.1%.\n\n\n```mermaid\nflowchart LR\n subgraph \"100% = SLO + Error Budget\"\n SLO[SLO: 99.9%
Must be good]:::green\n EB[Error Budget: 0.1%
Allowed to be bad]:::orange\n end\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n### How Error Budgets Work\n\nFor a 30-day window with 99.9% availability SLO:\n\n\n```shell\nTotal minutes in 30 days: 43,200\nError budget (0.1%): 43.2 minutes\n\nYou can be \"unavailable\" for 43.2 minutes per month\nwithout violating your SLO.\n```\n\n\n### Spending Error Budget\n\nError budget is \"spent\" when the SLI drops below the threshold:\n\n\n```mermaid\nflowchart TB\n subgraph \"Error Budget Consumers\"\n I[Incidents
Unplanned downtime]:::red\n R[Releases
Deployment issues]:::orange\n M[Maintenance
Planned downtime]:::yellow\n E[Experiments
Chaos testing]:::teal\n end\n\n EB[Error Budget]:::primary\n\n I --> EB\n R --> EB\n M --> EB\n E --> EB\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n```\n\n\n### Error Budget Policy\n\nAn error budget policy defines what happens when the budget is consumed:\n\n\n```shell\nError Budget Status:\n─────────────────────────────────────\n> 50% remaining: Normal development velocity\n Release at will, feature work prioritized\n\n25-50% remaining: Caution\n Risky releases require approval\n Some reliability work prioritized\n\n< 25% remaining: Freeze\n Only critical fixes and reliability work\n No new features until budget recovers\n\nExhausted: Emergency\n All hands on reliability\n Post-mortem required for any release\n```\n\n\n### Benefits of Error Budgets\n\n**1. Balances reliability and velocity**\n\nWithout error budgets, teams either:\n\n- Never release (too conservative)\n- Release constantly (ignore reliability)\n\nError budgets provide balance: release until you have spent your budget.\n\n**2. Makes trade-offs explicit**\n\n\"Can we ship this risky feature?\" becomes \"Do we have error budget?\"\n\n**3. Aligns teams**\n\nDevelopment wants features. Operations wants stability. Error budgets give both a shared framework.\n\n**4. Enables experimentation**\n\nIf you have budget, you can safely test things like:\n\n- Chaos engineering\n- Canary releases\n- New infrastructure\n\n---\n\n# Burn Rate Alerting\n\nTraditional alerting fires when a threshold is crossed. Burn rate alerting fires when error budget is being consumed too fast.\n\n### Burn Rate Concept\n\n\n```shell\nBurn rate = (Current error rate) / (Allowed error rate)\n\nIf SLO allows 0.1% errors over 30 days:\n- 0.1% error rate = burn rate 1x (sustainable)\n- 1.0% error rate = burn rate 10x (budget gone in 3 days)\n- 10% error rate = burn rate 100x (budget gone in 7 hours)\n```\n\n\n### Multi-Window Burn Rate\n\nUse multiple time windows for different severities:\n\n\n```mermaid\nflowchart TB\n subgraph \"Burn Rate Windows\"\n F[Fast burn
5 min window
10x rate]:::red\n M[Medium burn
1 hour window
5x rate]:::orange\n S[Slow burn
24 hour window
2x rate]:::yellow\n end\n\n F --> P1[Page immediately]:::red\n M --> P2[Page if persists]:::orange\n S --> T[Create ticket]:::green\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n\n| Scenario | Burn Rate | Time to Exhaust Budget | Alert |\n|----------|-----------|------------------------|-------|\n| Major outage | 100x | 7 hours | Page immediately |\n| Significant degradation | 10x | 3 days | Page after 5 min |\n| Slow degradation | 2x | 15 days | Ticket |\n| Normal operation | 1x | 30 days | None |\n\n\n### Why Burn Rate Is Better\n\nTraditional threshold alert:\n\n\n```shell\nAlert when error rate > 1%\n\nProblem: 0.9% errors for a week = SLO violated, no alert\n 1.1% errors for 1 minute = Alert, but SLO is fine\n```\n\n\nBurn rate alert:\n\n\n```shell\nAlert when burning budget 10x faster than sustainable\n\nCatches both:\n- Brief severe incidents (fast burn)\n- Prolonged minor issues (slow burn)\n```\n\n\n---\n\n# Service Level Agreements (SLAs)\n\nAn SLA is a formal contract with consequences for missing targets. SLAs are for external customers, not internal teams.\n\n### SLA vs SLO\n\n\n```mermaid\nflowchart LR\n subgraph Internal\n SLO[SLO: 99.95%
Target for team]:::primary\n end\n\n subgraph External\n SLA[SLA: 99.9%
Promise to customers]:::orange\n end\n\n SLO --> |Buffer| SLA\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n**Key difference:** SLOs are targets. SLAs are contracts.\n\n\n| Aspect | SLO | SLA |\n|--------|-----|-----|\n| **Audience** | Internal team | External customers |\n| **Consequences** | Policy enforcement | Financial penalties |\n| **Buffer** | No | Should be looser than SLO |\n| **Public** | Usually not | Yes, in contracts |\n\n\n### SLA Best Practices\n\n**1. Set SLA looser than SLO**\n\nIf your SLO is 99.95%, your SLA should be 99.9% or lower. This provides a buffer.\n\n**2. Define measurement clearly**\n\n\n```shell\nGood SLA definition:\n\"Availability is measured as the percentage of 1-minute intervals\nwhere the service responds to health checks within 5 seconds.\nExcluded: scheduled maintenance windows announced 48 hours in advance.\"\n```\n\n\n**3. Specify consequences**\n\n\n```shell\nService credits for availability below 99.9%:\n- 99.5% - 99.9%: 10% credit\n- 99.0% - 99.5%: 25% credit\n- Below 99.0%: 50% credit\n```\n\n\n**4. Include exclusions**\n\nSLAs typically exclude:\n\n- Scheduled maintenance\n- Customer-caused issues\n- Force majeure\n- Third-party dependencies\n\n### Real-World SLA Examples\n\n**AWS EC2:**\n\n- 99.99% uptime commitment\n- 10% credit for <99.99%\n- 30% credit for <99.0%\n\n**Google Cloud:**\n\n- 99.95% for most services\n- 10-50% credits for violations\n\n**Stripe API:**\n\n- 99.99% uptime\n- Credits for violations\n\n---\n\n# Implementing SLOs\n\n### Step 1: Identify User Journeys\n\nStart with what users actually do:\n\n\n```shell\nE-commerce example:\n─────────────────────────────────────\n1. Browse products (search, filter)\n2. View product details\n3. Add to cart\n4. Checkout and pay\n5. Track order\n\nEach journey has different importance and requirements.\n```\n\n\n### Step 2: Define SLIs for Each Journey\n\n\n```shell\nCheckout journey SLIs:\n─────────────────────────────────────\nAvailability: Percentage of checkout attempts that succeed\nLatency: Percentage of checkouts completing under 5 seconds\nQuality: Percentage of payments processed without retry\n```\n\n\n### Step 3: Set Initial SLOs\n\nBase on historical data plus business requirements:\n\n\n```shell\nHistorical performance (last 90 days):\n- Availability: 99.7%\n- Latency (p95): 3.2s\n\nBusiness requirement: Competitive with industry\n\nInitial SLOs:\n- Availability: 99.9% (stretch goal)\n- Latency: 95% under 5s (achievable with buffer)\n```\n\n\n### Step 4: Implement Measurement\n\n\n```shell\nPrometheus queries for checkout SLO:\n─────────────────────────────────────\n# Availability SLI\nsum(rate(checkout_total{status=\"success\"}[5m]))\n/ sum(rate(checkout_total[5m]))\n\n# Latency SLI\nsum(rate(checkout_duration_bucket{le=\"5.0\"}[5m]))\n/ sum(rate(checkout_duration_count[5m]))\n```\n\n\n### Step 5: Create Error Budget Dashboard\n\n\n```shell\nDashboard elements:\n─────────────────────────────────────\n- Current SLI value vs SLO target\n- Error budget remaining (percentage and minutes)\n- Error budget burn rate\n- Budget consumption over time\n- Days until budget exhaustion at current rate\n```\n\n\n### Step 6: Define Error Budget Policy\n\nDocument what happens at different budget levels.\n\n### Step 7: Iterate\n\nSLOs are not permanent. Adjust based on:\n\n- Actual user feedback\n- Changing business needs\n- System improvements\n- Historical achievement\n\n---\n\n# Summary\n\nSLIs, SLOs, and SLAs provide a framework for defining reliability:\n\n- **SLI (Indicator):** A metric measuring user experience\n - Must be user-centric, measurable, and specific\n - Common categories: availability, latency, throughput, quality\n- **SLO (Objective):** Target value for an SLI\n - Defines \"good enough\" reliability\n - Balance user expectations, cost, and business needs\n - Each additional nine is exponentially more expensive\n- **SLA (Agreement):** Contract with consequences\n - External commitment to customers\n - Should be looser than internal SLOs\n - Include clear measurement and exclusions\n\nError budgets make trade-offs explicit:\n\n- Error budget = 1 - SLO\n- Tracks how much unreliability is \"allowed\"\n- Balances reliability and development velocity\n- Error budget policy defines actions at different levels\n\nBurn rate alerting:\n\n- Alerts when budget is being consumed too fast\n- Better than static thresholds\n- Catches both severe incidents and slow degradation","pageType":"system-design"}

SLO, SLI, SLA

Ashish Pratap Singh

Get Premium