{"title":"Auto Scaling","description":"","content":"In 2015, Netflix saw a show premiere drive traffic to 3x normal in under 10 minutes. Auto-scaling launched hundreds of instances, capacity caught up, and nothing paged an engineer. The system simply scaled.\n\nThat is the promise of auto-scaling: infrastructure that expands and shrinks with demand. But getting it right requires more than turning it on. You need to understand the trade-offs, failure modes, and the choices that make scaling reliable.\n\nYou have estimated load, mapped traffic patterns, and calculated capacity. The next step is matching capacity to demand as it changes across hours, days, and seasons. Manual scaling is slow and error-prone. Auto-scaling automates the loop: add capacity as load rises, remove it as load falls.\n\nDone well, it turns capacity planning into a live system. You run close to average most of the time, then scale up for peaks automatically, making your earlier estimates and pattern analysis useful in production.\n\n---\n\n# Why Auto Scaling Matters\n\nConsider the cost and reliability implications of different scaling approaches:\n\n\n```mermaid\nflowchart TD\n Approach[Capacity Approach]:::primary\n\n Approach --> Fixed[Fixed at Peak
Capacity]:::orange\n Approach --> Average[Fixed at Average
Capacity]:::teal\n Approach --> Manual[Manual
Scaling]:::purple\n Approach --> Auto[Auto
Scaling]:::green\n\n Fixed --> FixedResult[High cost, high reliability
70% capacity wasted]:::orange\n Average --> AverageResult[Low cost, low reliability
Fails during peaks]:::teal\n Manual --> ManualResult[Variable cost, variable reliability
Requires constant attention]:::purple\n Auto --> AutoResult[Low cost, high reliability
Scales with demand]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n\n| Approach | Cost | Reliability | Operational Burden | Best For |\n|----------|------|-------------|-------------------|----------|\n| **Fixed at peak capacity** | High (pay for unused) | High (always ready) | Low | Simple, critical systems |\n| **Fixed at average capacity** | Medium | Low (fails at peaks) | Low | Uniform traffic only |\n| **Manual scaling** | Variable | Variable | High | Infrequent changes |\n| **Auto scaling** | Low (pay for use) | High (matches demand) | Low (after setup) | Variable traffic patterns |\n\n\nWithout auto-scaling, you face an uncomfortable trade-off between cost and reliability. Auto-scaling breaks this trade-off by dynamically adjusting capacity.\n\n### The Economics of Auto Scaling\n\nFor a system with significant daily variation, the savings are substantial:\n\n\n```shell\nExample: Service with 3x peak-to-average ratio\n\nWithout auto-scaling (provisioned for peak):\n- Peak capacity: 30 servers running 24/7\n- Average utilization: 33% (wasted 67% of the time)\n- Monthly cost: $30,000\n\nWith auto-scaling:\n- Peak: 30 servers (2 hours/day during evening peak)\n- Normal: 15 servers (8 hours/day)\n- Low: 8 servers (14 hours/day overnight)\n- Weighted average: 12 servers\n- Monthly cost: $12,000 + scaling overhead (~$1,000)\n\nAnnual savings: $204,000 (57% reduction)\n```\n\n\nThe actual savings depend on traffic patterns, scaling responsiveness, and minimum capacity requirements. Applications with high peak-to-average ratios (5x+) can see 60-70% savings. Applications with lower ratios (2x) might see 20-30% savings. But even modest savings compound significantly over time and at scale.\n\n\n```mermaid\nflowchart LR\n subgraph Without[\"Without Auto-Scaling\"]\n W1[Peak Capacity
30 servers]:::orange\n W2[Average Load
10 servers worth]:::teal\n W3[Wasted
20 servers]:::red\n end\n\n subgraph With[\"With Auto-Scaling\"]\n A1[Peak Hours
30 servers]:::orange\n A2[Normal Hours
15 servers]:::teal\n A3[Low Hours
8 servers]:::green\n end\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n### Beyond Cost Savings\n\nAuto-scaling provides benefits beyond cost reduction:\n\n\n| Benefit | How |\n|---------|-----|\n| **Reliability** | Automatically adds capacity before overload |\n| **Resilience** | Replaces failed instances automatically |\n| **Operational simplicity** | No manual intervention for routine scaling |\n| **Consistent performance** | Maintains target metrics across load levels |\n| **Event handling** | Scales for known events without manual work |\n\n\n---\n\n# How Auto Scaling Works\n\nAuto-scaling is a feedback control loop. It continuously monitors system metrics, compares them to desired targets, and adjusts capacity to close the gap.\n\n\n```mermaid\nflowchart TD\n subgraph Loop[\"Auto-Scaling Control Loop\"]\n Metrics[1. Collect
Metrics]:::primary\n Compare[2. Compare to
Target]:::orange\n Decide[3. Scaling
Decision]:::teal\n Action[4. Execute
Action]:::green\n Stabilize[5. Wait for
Stabilization]:::purple\n end\n\n Metrics --> Compare\n Compare --> Decide\n Decide --> Action\n Action --> Stabilize\n Stabilize --> Metrics\n\n Decide --> |Above target| ScaleOut[Scale Out]:::green\n Decide --> |Below target| ScaleIn[Scale In]:::orange\n Decide --> |Within range| NoChange[No Change]:::teal\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n### The Five Stages of Auto-Scaling\n\n#### **Stage 1: Metrics Collection**\n\nAuto-scaling decisions are based on metrics. The system continuously collects data from various sources:\n\n\n| Metric Source | Examples | Update Frequency |\n|---------------|----------|------------------|\n| **Infrastructure** | CPU, memory, network I/O | Every 1-5 minutes |\n| **Load balancer** | Request count, latency, error rate | Every 1 minute |\n| **Application** | Queue depth, active connections, custom | Real-time to 1 minute |\n| **External** | Scheduled events, business metrics | Varies |\n\n\n#### **Stage 2: Comparison to Target**\n\nThe collected metrics are compared against configured targets or thresholds:\n\n\n```shell\nCurrent: CPU at 85%\nTarget: CPU at 70%\nGap: 15% above target → Scale out needed\n```\n\n\n#### **Stage 3: Scaling Decision**\n\nBased on the comparison and configured policies, the system decides whether to scale:\n\n\n| Condition | Decision |\n|-----------|----------|\n| Metric above scale-out threshold for required duration | Scale out |\n| Metric below scale-in threshold for required duration | Scale in |\n| Metric within acceptable range | No change |\n| Cooldown period active | No change (wait) |\n\n\n#### **Stage 4: Execute Action**\n\nThe scaling action is executed:\n\n\n| Action | Description | Typical Time |\n|--------|-------------|--------------|\n| **Launch instance** | Start new VM/container | 1-5 min (VM), 10-30s (container) |\n| **Register with LB** | Add to load balancer | 10-30 seconds |\n| **Health check pass** | Verify instance is healthy | 30-60 seconds |\n| **Receive traffic** | Start handling requests | Immediate after registration |\n| **Terminate instance** | Stop and release resources | 10-30 seconds |\n\n\n#### **Stage 5: Stabilization**\n\nAfter scaling, the system waits during a cooldown period to let metrics stabilize before making additional decisions. This prevents oscillation.\n\n### End-to-End Scaling Timeline\n\nUnderstanding the total time from trigger to traffic handling is critical for capacity planning:\n\n\n```mermaid\nflowchart LR\n subgraph Timeline[\"Scale-Out Timeline (VM)\"]\n T0[Trigger
0:00]:::red\n T1[Launch
0:00-3:00]:::orange\n T2[Boot
3:00-4:00]:::orange\n T3[App Start
4:00-5:00]:::orange\n T4[Health Check
5:00-5:30]:::teal\n T5[Receiving Traffic
5:30+]:::green\n end\n\n T0 --> T1 --> T2 --> T3 --> T4 --> T5\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n\n| Environment | Trigger to Traffic | Key Bottleneck |\n|-------------|-------------------|----------------|\n| **VMs (AWS EC2)** | 3-7 minutes | Instance launch and boot |\n| **Containers (ECS, GKE)** | 30s-2 minutes | Image pull, container start |\n| **Kubernetes pods** | 10s-1 minute | Scheduler, image pull |\n| **Serverless (Lambda)** | 100ms-1s (warm), 1-10s (cold) | Cold start initialization |\n\n\nThis timeline has critical implications: if traffic can spike 10x in 2 minutes, VM-based auto-scaling will not respond fast enough. You need either containers for faster scaling, serverless for instant scaling, or sufficient buffer capacity to survive while scaling occurs.\n\n---\n\n# Auto Scaling Strategies\n\nDifferent strategies suit different traffic patterns and requirements. Most production systems combine multiple strategies for comprehensive coverage.\n\n### Reactive (Target Tracking) Scaling\n\nThe most common approach: maintain a target metric value by adding or removing capacity as needed.\n\n\n```mermaid\nflowchart LR\n subgraph Reactive[\"Target Tracking Scaling\"]\n Current[Current
85% CPU]:::red\n Target[Target
70% CPU]:::green\n Gap[Gap
+15%]:::orange\n Action[Add Capacity
Until Target Met]:::teal\n end\n\n Current --> Gap\n Target --> Gap\n Gap --> Action\n Action -.-> |Reduces| Current\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### **How it works**\n\n1. Set target value (e.g., 70% CPU utilization)\n2. System continuously compares current to target\n3. If above target: calculate and add needed capacity\n4. If below target: calculate and remove excess capacity\n5. Adjustments are proportional to the gap\n\n#### **Configuration example (AWS Target Tracking)**\n\n\n```json\n{\n \"PolicyName\": \"cpu-target-tracking\",\n \"PolicyType\": \"TargetTrackingScaling\",\n \"TargetTrackingConfiguration\": {\n \"PredefinedMetricSpecification\": {\n \"PredefinedMetricType\": \"ASGAverageCPUUtilization\"\n },\n \"TargetValue\": 70.0,\n \"ScaleOutCooldown\": 60,\n \"ScaleInCooldown\": 300,\n \"DisableScaleIn\": false\n }\n}\n```\n\n\n\n| Pros | Cons |\n|------|------|\n| Simple to configure (just set target) | Reactive (responds after load increases) |\n| Self-adjusting without tuning | Scaling lag during rapid spikes |\n| Works for most steady workloads | May oscillate around target |\n| Cloud-managed, minimal maintenance | Single metric may miss complexity |\n\n\n**Best for:** Steady traffic with gradual changes, CPU or memory-bound workloads, teams wanting simple configuration.\n\n### Step Scaling\n\nDefine explicit scaling steps based on metric thresholds. Allows different responses to different severity levels.\n\n\n```mermaid\nflowchart TD\n CPU[CPU Utilization]:::primary\n\n CPU --> Low[0-50%
Minimum: 5 instances]:::green\n CPU --> Normal[50-70%
Base: 10 instances]:::teal\n CPU --> Elevated[70-80%
Add 5 instances]:::orange\n CPU --> High[80-90%
Add 10 instances]:::orange\n CPU --> Critical[>90%
Add 20 instances]:::red\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n#### **Configuration example (AWS Step Scaling)**\n\n\n```json\n{\n \"PolicyName\": \"cpu-step-scaling\",\n \"PolicyType\": \"StepScaling\",\n \"AdjustmentType\": \"ChangeInCapacity\",\n \"StepAdjustments\": [\n {\n \"MetricIntervalLowerBound\": 0,\n \"MetricIntervalUpperBound\": 10,\n \"ScalingAdjustment\": 2\n },\n {\n \"MetricIntervalLowerBound\": 10,\n \"MetricIntervalUpperBound\": 20,\n \"ScalingAdjustment\": 5\n },\n {\n \"MetricIntervalLowerBound\": 20,\n \"ScalingAdjustment\": 10\n }\n ],\n \"Cooldown\": 60\n}\n```\n\n\nThe bounds are relative to the alarm threshold. If the alarm triggers at 70% CPU:\n\n- 70-80% CPU: Add 2 instances\n- 80-90% CPU: Add 5 instances\n- 90%+ CPU: Add 10 instances\n\n\n| Pros | Cons |\n|------|------|\n| Aggressive response to high load | More complex to configure |\n| Different actions for different severity | Requires understanding of workload |\n| Can scale faster than target tracking | Step boundaries may not match traffic |\n| Fine-grained control | More parameters to tune |\n\n\n**Best for:** Traffic with sudden spikes, systems where fast scale-out is critical, workloads with known load levels.\n\n### Scheduled Scaling\n\nScale based on time, not metrics. Proactively provision capacity for predictable patterns.\n\n\n```mermaid\nflowchart TB\n subgraph Day[\"Weekday Schedule\"]\n S1[6 AM
Scale: 20]:::teal\n S2[9 AM
Scale: 50]:::orange\n S3[12 PM
Scale: 40]:::orange\n S4[6 PM
Scale: 80]:::red\n S5[10 PM
Scale: 40]:::orange\n S6[2 AM
Scale: 10]:::teal\n end\n\n S1 --> S2 --> S3 --> S4 --> S5 --> S6\n\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Configuration example:**\n\n\n```json\n{\n \"ScheduledActions\": [\n {\n \"ScheduledActionName\": \"morning-scale-up\",\n \"Recurrence\": \"0 6 * * MON-FRI\",\n \"MinSize\": 20,\n \"MaxSize\": 100,\n \"DesiredCapacity\": 20\n },\n {\n \"ScheduledActionName\": \"business-hours\",\n \"Recurrence\": \"0 9 * * MON-FRI\",\n \"DesiredCapacity\": 50\n },\n {\n \"ScheduledActionName\": \"evening-peak\",\n \"Recurrence\": \"0 18 * * *\",\n \"DesiredCapacity\": 80\n },\n {\n \"ScheduledActionName\": \"night-scale-down\",\n \"Recurrence\": \"0 2 * * *\",\n \"DesiredCapacity\": 10\n }\n ]\n}\n```\n\n\n\n| Pros | Cons |\n|------|------|\n| Proactive (capacity ready before load) | Requires accurate pattern knowledge |\n| No lag between load and capacity | Does not adapt to unexpected changes |\n| Predictable capacity and cost | Must update as patterns change |\n| Works when metrics are delayed | Only useful for predictable patterns |\n\n\n**Best for:** Known daily patterns, B2B apps with business-hours traffic, planned events like product launches or marketing campaigns.\n\n### Predictive Scaling\n\nUses machine learning to forecast load and scale proactively. Combines the benefits of scheduled and reactive scaling.\n\n\n```mermaid\nflowchart LR\n Historical[14+ Days
Historical Data]:::primary --> ML[ML Model
Pattern Detection]:::orange\n ML --> Forecast[Traffic
Forecast]:::teal\n Forecast --> PreScale[Pre-Provision
Capacity]:::green\n Actual[Actual Traffic]:::purple --> Compare{Compare}\n Forecast --> Compare\n Compare --> |Adjust| ML\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n#### **How it works**\n\n1. Analyzes 14+ days of historical load data\n2. Identifies patterns (daily cycles, weekly patterns)\n3. Forecasts future load for the next 48 hours\n4. Pre-provisions capacity before predicted peaks\n5. Continuously refines predictions based on actual traffic\n\n#### **AWS Predictive Scaling configuration**\n\n\n```json\n{\n \"PolicyName\": \"predictive-scaling\",\n \"PolicyType\": \"PredictiveScaling\",\n \"PredictiveScalingConfiguration\": {\n \"MetricSpecifications\": [{\n \"TargetValue\": 70,\n \"PredefinedMetricPairSpecification\": {\n \"PredefinedMetricType\": \"ASGCPUUtilization\"\n }\n }],\n \"Mode\": \"ForecastAndScale\",\n \"SchedulingBufferTime\": 300,\n \"MaxCapacityBreachBehavior\": \"IncreaseMaxCapacity\"\n }\n}\n```\n\n\n\n| Pros | Cons |\n|------|------|\n| Combines proactive and reactive | Requires 14+ days of data |\n| Adapts to changing patterns | May not predict unusual events |\n| Handles both daily and weekly cycles | Cloud-specific implementation |\n| Less manual configuration than scheduled | Model training takes time |\n\n\n**Best for:** Applications with regular patterns and occasional unexpected traffic, teams wanting automation without manual schedule management.\n\n### Combining Strategies\n\nReal-world systems often layer multiple strategies for comprehensive coverage:\n\n\n```mermaid\nflowchart LR\n Traffic[Incoming Traffic]:::primary\n\n Traffic --> Scheduled[Layer 1: Scheduled
Pre-scale for known patterns]:::teal\n Scheduled --> Predictive[Layer 2: Predictive
Adjust for predicted variations]:::orange\n Predictive --> Reactive[Layer 3: Reactive
Handle unexpected load]:::green\n Reactive --> Step[Layer 4: Step Scaling
Emergency response to spikes]:::red\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n\n| Layer | Strategy | Purpose | Example |\n|-------|----------|---------|---------|\n| **Base** | Scheduled | Ensure capacity for known patterns | Scale to 50 at 9 AM weekdays |\n| **Adaptive** | Predictive | Adjust for predicted variations | ML-based daily adjustments |\n| **Reactive** | Target tracking | Handle variations around baseline | Maintain 70% CPU |\n| **Emergency** | Step scaling | Rapid response to severe spikes | Add 20 instances if CPU > 90% |\n\n\n---\n\n# Choosing Scaling Metrics\n\nThe metric you scale on determines how well auto-scaling matches your actual capacity needs. Choosing the wrong metric leads to either over-scaling (wasted cost) or under-scaling (performance degradation).\n\n### Common Metrics Comparison\n\n\n```mermaid\nflowchart TD\n subgraph Metrics[\"Scaling Metric Selection\"]\n CPU[CPU Utilization]:::primary\n Memory[Memory Usage]:::primary\n Requests[Request Count]:::primary\n Queue[Queue Depth]:::primary\n Latency[Response Latency]:::primary\n Custom[Custom Metrics]:::primary\n end\n\n CPU --> CPUUse[CPU-bound workloads
Compute-heavy processing]:::teal\n Memory --> MemUse[Memory-bound workloads
Caching, in-memory processing]:::teal\n Requests --> ReqUse[Web servers, APIs
Known requests per instance]:::teal\n Queue --> QueueUse[Async workers
Background processing]:::teal\n Latency --> LatUse[User-facing services
SLA-driven applications]:::teal\n Custom --> CustomUse[Complex workloads
Specific bottlenecks]:::teal\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n\n| Metric | Response Time | Accuracy | Best For | Watch Out For |\n|--------|---------------|----------|----------|---------------|\n| **CPU utilization** | Fast | High for CPU-bound | Compute workloads | I/O-bound apps show low CPU |\n| **Memory utilization** | Slow | Medium | Memory-intensive | Memory grows slowly, scales late |\n| **Request count** | Fast | High | Web servers, APIs | Does not account for request complexity |\n| **Queue depth** | Fast | High | Async workers | Queue may drain quickly |\n| **Response latency** | Medium | High for UX | Customer-facing | Latency spikes may be transient |\n| **Active connections** | Fast | Medium | Connection-limited | Connections may persist after work done |\n| **Custom metrics** | Varies | High for specific | Complex workloads | Requires instrumentation |\n\n\n### CPU Utilization\n\nThe most common metric, but not always the best choice.\n\n**When CPU is appropriate:**\n\n- Application performance scales linearly with CPU\n- Response time correlates with CPU usage\n- No significant I/O waits or external dependencies\n- Compute-heavy workloads (image processing, calculations)\n\n**When CPU is problematic:**\n\n- I/O-bound applications (waiting on database, network)\n- Memory-bound applications (caching, in-memory processing)\n- Applications with external bottlenecks (third-party APIs)\n- Workloads with high I/O wait (disk, network)\n\n\n```shell\nExample: I/O-bound application\n\nRequest flow:\n1. Receive request (1ms)\n2. Query database (50ms wait)\n3. Process result (2ms)\n4. Return response (1ms)\n\nTotal: 54ms, but only 4ms is CPU work\nCPU utilization: ~7% even under full load\nScaling on CPU: Would never scale despite overload\n```\n\n\nFor I/O-bound applications, request-based or latency-based scaling is more appropriate.\n\n### Request-Based Scaling\n\nScale based on request count per instance. Directly tied to workload.\n\n\n```shell\nConfiguration:\n- Target: 1,000 requests per instance per minute\n- Current: 3,000 requests total, 2 instances\n- Current per instance: 1,500 requests\n- Action: Add 1 instance (3,000 / 3 = 1,000 per instance)\n```\n\n\n**AWS Application Load Balancer target tracking:**\n\n\n```json\n{\n \"TargetTrackingConfiguration\": {\n \"PredefinedMetricSpecification\": {\n \"PredefinedMetricType\": \"ALBRequestCountPerTarget\",\n \"ResourceLabel\": \"app/my-alb/50dc6c495c0c9188/targetgroup/my-target-group/cbf133c568e0d028\"\n },\n \"TargetValue\": 1000\n }\n}\n```\n\n\n**Advantages:**\n\n- Directly tied to actual workload\n- Works regardless of where time is spent (CPU, I/O, waiting)\n- Predictable capacity per request\n- Easy to reason about\n\n**Disadvantages:**\n\n- Does not account for request complexity (some requests may be 10x heavier)\n- Requires knowing sustainable requests per instance\n\n### Queue-Based Scaling\n\nFor async processing, scale based on queue depth or age of oldest message.\n\n\n```mermaid\nflowchart LR\n Queue[(Queue
5,000 messages)]:::orange --> Workers[Workers
5 instances]:::teal\n\n Queue --> |Depth > 1000| ScaleUp[Add Workers]:::green\n Queue --> |Depth < 100| ScaleDown[Remove Workers]:::red\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Calculating workers needed:**\n\n\n```shell\nGoal: Process queue within acceptable time\n\nQueue depth: 10,000 messages\nTarget processing time: 5 minutes\nProcessing rate: 100 messages/minute/worker\n\nWorkers needed = Queue depth / (Processing rate × Target time)\nWorkers needed = 10,000 / (100 × 5) = 20 workers\n\nWith buffer for efficiency losses:\nWorkers needed = 20 × 1.2 = 24 workers\n```\n\n\n**KEDA (Kubernetes Event-Driven Autoscaling) example:**\n\n\n```yaml\napiVersion: keda.sh/v1alpha1\nkind: ScaledObject\nmetadata:\n name: queue-worker-scaler\nspec:\n scaleTargetRef:\n name: queue-worker\n minReplicaCount: 1\n maxReplicaCount: 50\n triggers:\n - type: aws-sqs-queue\n metadata:\n queueURL: https://sqs.us-east-1.amazonaws.com/123456789/my-queue\n queueLength: \"500\"\n awsRegion: us-east-1\n```\n\n\n### Custom Metrics\n\nWhen standard metrics do not capture your bottleneck, create custom metrics:\n\n\n| Custom Metric | Use Case | How to Implement |\n|---------------|----------|------------------|\n| **Database connection pool usage** | Database-bound apps | Publish pool utilization to CloudWatch |\n| **Requests in flight** | Rate-limited dependencies | Count active requests, publish metric |\n| **Business metrics** | Orders pending, users waiting | Application-level instrumentation |\n| **Cache hit rate** | Cache-dependent performance | Cache client metrics |\n| **External API quota usage** | Third-party API limits | Track API call counts |\n\n\n**Example: Custom metric for database connection pool**\n\n\n```python\n# Publish custom metric from application\nimport boto3\n\ncloudwatch = boto3.client('cloudwatch')\n\ndef publish_connection_pool_metric(pool_size, active_connections):\n utilization = (active_connections / pool_size) * 100\n cloudwatch.put_metric_data(\n Namespace='MyApp/Database',\n MetricData=[{\n 'MetricName': 'ConnectionPoolUtilization',\n 'Value': utilization,\n 'Unit': 'Percent'\n }]\n )\n```\n\n\nThen configure auto-scaling to use this metric:\n\n\n```json\n{\n \"TargetTrackingConfiguration\": {\n \"CustomizedMetricSpecification\": {\n \"MetricName\": \"ConnectionPoolUtilization\",\n \"Namespace\": \"MyApp/Database\",\n \"Statistic\": \"Average\"\n },\n \"TargetValue\": 60\n }\n}\n```\n\n\n---\n\n# Setting Thresholds and Cooldowns\n\nGetting thresholds and timing right prevents both under-scaling (capacity shortage) and thrashing (constant scaling up and down).\n\n### Threshold Guidelines\n\n\n```mermaid\nflowchart LR\n subgraph Thresholds[\"Asymmetric Thresholds\"]\n ScaleOut[Scale Out
70%]:::red\n Target[Target
60%]:::orange\n ScaleIn[Scale In
40%]:::green\n end\n\n ScaleOut --> |\"Wide gap prevents oscillation\"| ScaleIn\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n\n| Metric | Scale Out Threshold | Scale In Threshold | Rationale |\n|--------|--------------------|--------------------|-----------|\n| **CPU** | > 70% | < 30% | Leave headroom for spikes |\n| **Memory** | > 75% | < 40% | Avoid OOM, memory is slow to release |\n| **Request rate** | > 1000/instance | < 400/instance | Wide gap prevents thrashing |\n| **Queue depth** | > 500 | < 50 | Process backlog quickly |\n| **Latency P95** | > 200ms | < 80ms | Maintain user experience |\n\n\n**Why asymmetric thresholds:**\n\nScale-out thresholds should be closer to target than scale-in thresholds. This asymmetry prevents oscillation:\n\n\n```shell\nBad configuration (symmetric):\n- Scale out at 70%\n- Scale in at 60%\n- Result: Traffic at 65% causes constant scaling\n\nGood configuration (asymmetric):\n- Scale out at 70%\n- Scale in at 40%\n- Result: Stable at any load between 40-70%\n```\n\n\n### Cooldown Periods\n\nCooldown periods prevent rapid, unnecessary scaling by requiring time between scaling events.\n\n\n```mermaid\nflowchart LR\n Trigger[Scale-Out
Triggered]:::red --> Execute[Add
Instances]:::orange\n Execute --> Warmup[Instance
Warmup]:::teal\n Warmup --> Cooldown[Cooldown
Period]:::purple\n Cooldown --> Monitor[Resume
Monitoring]:::green\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n\n| Cooldown Type | Purpose | Typical Duration | Rationale |\n|---------------|---------|------------------|-----------|\n| **Scale-out cooldown** | Let new instances warm up | 60-180 seconds | Metrics need time to reflect new capacity |\n| **Scale-in cooldown** | Ensure load is truly decreasing | 300-600 seconds | Avoid removing capacity prematurely |\n\n\n**Why asymmetric cooldowns:**\n\n\n| Situation | Risk | Mitigation |\n|-----------|------|------------|\n| **Too-fast scale out** | Extra cost, usually acceptable | Short cooldown (1-3 min) |\n| **Too-fast scale in** | Capacity shortage, user impact | Longer cooldown (5-10 min) |\n\n\nScaling out too quickly costs money but rarely causes problems. Scaling in too quickly removes capacity users need and causes degradation. Error on the side of keeping capacity.\n\n### Evaluation Period\n\nHow long a condition must persist before triggering scaling:\n\n\n```shell\nScale-out policy:\n- CPU > 70% for 2 consecutive 1-minute periods (2 minutes total)\n\nScale-in policy:\n- CPU < 40% for 10 consecutive 1-minute periods (10 minutes total)\n```\n\n\n\n| Evaluation Approach | Pros | Cons |\n|---------------------|------|------|\n| **Short (1-2 minutes)** | Fast response | May react to noise |\n| **Medium (3-5 minutes)** | Balanced | Slight delay |\n| **Long (10+ minutes)** | Ignores transients | Slow response to real changes |\n\n\nLonger evaluation periods for scale-in prevent removing capacity for brief load dips. Shorter periods for scale-out enable quick response to increasing load.\n\n---\n\n# Auto Scaling in Different Environments\n\nAuto-scaling works differently across VMs, containers, and serverless environments. Understanding these differences helps you choose the right approach.\n\n### VM-Based Auto Scaling (EC2, Compute Engine)\n\n\n```mermaid\nflowchart TD\n ASG[Auto Scaling Group]:::primary\n\n ASG --> Launch[Launch Template
AMI, instance type, config]:::orange\n ASG --> Policies[Scaling Policies
When to scale]:::teal\n ASG --> Health[Health Checks
Replace unhealthy]:::green\n ASG --> LB[Load Balancer
Traffic distribution]:::purple\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n#### **Key characteristics**\n\n- Slowest scaling (2-5 minutes to launch and serve traffic)\n- Highest per-instance overhead\n- Full control over instance configuration\n- Best for stateful or complex applications\n\n#### **Best practices**\n\n- Use smaller AMIs for faster boot\n- Consider warm pools for pre-launched instances\n- Multi-AZ deployment for resilience\n- Instance refresh for updates\n\n#### **Warm pools (pre-launched instances)**\n\n\n```json\n{\n \"WarmPool\": {\n \"MinSize\": 5,\n \"MaxGroupPreparedCapacity\": 10,\n \"PoolState\": \"Stopped\",\n \"InstanceReusePolicy\": {\n \"ReuseOnScaleIn\": true\n }\n }\n}\n```\n\n\nWarm pools keep instances in stopped state, ready to start quickly. Reduces scaling time from 3-5 minutes to 1-2 minutes.\n\n### Container-Based Auto Scaling (ECS, GKE, EKS)\n\nContainers add layers of scaling: both the containers (tasks/pods) and the underlying nodes can scale.\n\n\n```mermaid\nflowchart LR\n subgraph ContainerScaling[\"Container Scaling Layers\"]\n HPA[Container/Pod
Autoscaler]:::primary\n Cluster[Cluster/Node
Autoscaler]:::orange\n end\n\n HPA --> |\"Scales\"| Pods[Pods/Tasks
10s-1min]:::green\n Cluster --> |\"Scales\"| Nodes[Nodes/Instances
2-5min]:::teal\n\n Pods --> |\"Need more
resources\"| Cluster\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### **Kubernetes Horizontal Pod Autoscaler (HPA)**\n\n\n```yaml\napiVersion: autoscaling/v2\nkind: HorizontalPodAutoscaler\nmetadata:\n name: web-app-hpa\nspec:\n scaleTargetRef:\n apiVersion: apps/v1\n kind: Deployment\n name: web-app\n minReplicas: 3\n maxReplicas: 50\n metrics:\n - type: Resource\n resource:\n name: cpu\n target:\n type: Utilization\n averageUtilization: 70\n - type: Resource\n resource:\n name: memory\n target:\n type: Utilization\n averageUtilization: 80\n behavior:\n scaleDown:\n stabilizationWindowSeconds: 300\n policies:\n - type: Percent\n value: 10\n periodSeconds: 60\n scaleUp:\n stabilizationWindowSeconds: 0\n policies:\n - type: Percent\n value: 100\n periodSeconds: 15\n```\n\n\n#### **Cluster Autoscaler**\n\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: cluster-autoscaler-priority-expander\ndata:\n priorities: |-\n 10:\n - .*spot.*\n 50:\n - .*on-demand.*\n```\n\n\n\n| Scaler | What It Scales | Scaling Speed | Based On |\n|--------|----------------|---------------|----------|\n| **HPA** | Pod replicas | 15s-1min | CPU, memory, custom metrics |\n| **VPA** | Pod resource requests | Requires restart | Historical usage |\n| **Cluster Autoscaler** | Cluster nodes | 2-5min | Pending pods |\n| **Karpenter** | Cluster nodes | 30s-2min | Pending pods, optimized |\n| **KEDA** | Pods | 15s-1min | External events (queues, etc.) |\n\n\n### Serverless Auto Scaling (Lambda, Cloud Functions)\n\nServerless platforms handle scaling automatically, but understanding the model helps optimize performance and cost.\n\n\n```mermaid\nflowchart LR\n Request[Request]:::primary --> Dispatcher[Dispatcher]:::orange\n\n Dispatcher --> |\"Warm instance
available\"| Warm[Warm Instance
<100ms]:::green\n Dispatcher --> |\"No warm
instance\"| Cold[Cold Start
100ms-10s]:::red\n\n Warm --> Response[Response]:::teal\n Cold --> Response\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### **Serverless scaling characteristics**\n\n\n| Aspect | Behavior | Implication |\n|--------|----------|-------------|\n| **Scaling speed** | Near-instant (milliseconds) | No need for predictive scaling |\n| **Cold starts** | 100ms-10s for new instances | May impact latency-sensitive apps |\n| **Concurrency limits** | Per-function and account limits | Can throttle under high load |\n| **Automatic scaling** | No configuration needed | Less control |\n\n\n#### **Provisioned concurrency (AWS Lambda)**\n\n\n```json\n{\n \"FunctionName\": \"my-function\",\n \"ProvisionedConcurrencyConfig\": {\n \"ProvisionedConcurrentExecutions\": 100\n }\n}\n```\n\n\nProvisioned concurrency keeps instances warm, eliminating cold starts at the cost of constant charges.\n\n#### **Reserved concurrency**\n\n\n```json\n{\n \"FunctionName\": \"critical-function\",\n \"ReservedConcurrentExecutions\": 500\n}\n```\n\n\nReserves capacity for critical functions, preventing them from being throttled when account limits are hit.\n\n---\n\n# Common Challenges and Solutions\n\n### Challenge 1: Slow Scaling Response\n\n**Problem:** Instances take too long to launch and warm up, leaving the system overloaded during scaling.\n\n\n```mermaid\nflowchart LR\n subgraph Problem[\"Slow Scaling Problem\"]\n Spike[Traffic
Spike]:::red --> Gap[Capacity
Gap]:::orange --> Degradation[Performance
Degradation]:::red\n Gap --> Scaling[Scaling
In Progress]:::teal\n end\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### **Solutions**\n\n\n| Solution | How It Helps | Trade-off |\n|----------|--------------|-----------|\n| **Smaller AMIs** | Faster boot time | Less pre-installed, more startup config |\n| **Warm pools** | Pre-launched stopped instances | Extra cost for idle instances |\n| **Containers** | 10-60s vs 3-5min for VMs | Different operational model |\n| **Buffer capacity** | Maintain 20-30% headroom | Higher baseline cost |\n| **Predictive scaling** | Scale before load arrives | Requires historical data |\n| **Faster health checks** | Quicker traffic routing | May route to unready instances |\n\n\n#### **Warm pool configuration**\n\n\n```json\n{\n \"WarmPool\": {\n \"MinSize\": 10,\n \"PoolState\": \"Stopped\",\n \"InstanceReusePolicy\": {\n \"ReuseOnScaleIn\": true\n }\n }\n}\n```\n\n\n### Challenge 2: Thrashing (Scale Oscillation)\n\n**Problem:** System repeatedly scales up and down, wasting resources and causing instability.\n\n\n```mermaid\nflowchart LR\n Up[Scale Up]:::green --> High[Load Drops
Below Threshold]:::orange\n High --> Down[Scale Down]:::red\n Down --> Low[Load Rises
Above Threshold]:::orange\n Low --> Up\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n#### **Causes and solutions**\n\n\n| Cause | Solution |\n|-------|----------|\n| Thresholds too close | Wider gap between scale-out and scale-in (70%/40% not 70%/60%) |\n| Cooldowns too short | Increase cooldowns, especially scale-in (5-10 min) |\n| Metric too volatile | Use averaged metrics over longer periods |\n| Traffic naturally oscillates | Accept some oscillation or use predictive scaling |\n\n\n### Challenge 3: Cold Start Performance\n\n**Problem:** New instances perform poorly before warming up, causing degraded performance even after scaling.\n\n**Impact:**\n\n- Cache is empty (all requests hit database)\n- JIT not warmed (Java/JVM applications slower initially)\n- Connection pools not established\n- DNS cache cold\n\n**Solutions:**\n\n\n| Solution | Implementation | Best For |\n|----------|----------------|----------|\n| **Slow start in LB** | Start with low weight, ramp up | All applications |\n| **Extended health checks** | Longer before receiving traffic | Applications with long init |\n| **Synthetic warm-up traffic** | Send test requests before live traffic | Cache-dependent apps |\n| **Cache pre-population** | Load hot data on startup | Cache-heavy applications |\n| **Connection pre-warming** | Establish DB connections at startup | Database-bound apps |\n\n\n**AWS ALB slow start configuration:**\n\n\n```json\n{\n \"TargetGroupAttributes\": [\n {\n \"Key\": \"slow_start.duration_seconds\",\n \"Value\": \"120\"\n }\n ]\n}\n```\n\n\nDuring slow start, the load balancer gradually increases traffic to new instances over the specified duration.\n\n### Challenge 4: Database Connection Exhaustion\n\n**Problem:** Scaling app servers exhausts database connections.\n\n\n```shell\nBefore scaling: 10 servers × 20 connections = 200 connections\nAfter scaling: 50 servers × 20 connections = 1,000 connections\nDatabase limit: 500 connections\nResult: Connection errors, cascading failures\n```\n\n\n\n```mermaid\nflowchart TD\n App1[App Server 1
20 connections]:::teal\n App2[App Server 2
20 connections]:::teal\n AppN[App Server 50
20 connections]:::teal\n\n App1 --> DB[(Database
Max: 500)]:::red\n App2 --> DB\n AppN --> DB\n\n Total[Total: 1000
connections needed]:::orange --> DB\n\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n**Solutions:**\n\n\n| Solution | How It Helps | Trade-off |\n|----------|--------------|-----------|\n| **Connection pooler (PgBouncer, ProxySQL)** | Multiplexes connections, 10:1 or higher ratio | Additional component |\n| **Smaller pools per instance** | Fewer connections per instance | Longer wait times |\n| **Connection-aware scaling** | Include DB connections in scaling metrics | More complex metrics |\n| **Database scaling** | Read replicas, larger instance | Higher database cost |\n| **Serverless database** | Auto-scales connections | Different pricing model |\n\n\n**PgBouncer configuration example:**\n\n\n```shell\n[databases]\nmydb = host=primary.db.example.com port=5432 dbname=mydb\n\n[pgbouncer]\npool_mode = transaction\nmax_client_conn = 10000\ndefault_pool_size = 20\nmin_pool_size = 5\n```\n\n\n### Challenge 5: Cascading Failures\n\n**Problem:** Scaling events trigger failures in dependent services.\n\n\n```mermaid\nflowchart TD\n Scale[App Servers
Scale Out]:::primary --> DBLoad[Increased
DB Load]:::orange\n DBLoad --> DBSlow[Database
Slows Down]:::red\n DBSlow --> Timeout[App
Timeouts]:::red\n Timeout --> MoreScale[More Scaling
Triggered]:::orange\n MoreScale --> DBLoad\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Solutions:**\n\n\n| Solution | Implementation |\n|----------|----------------|\n| **Circuit breakers** | Stop calling overwhelmed dependencies |\n| **Rate limiting** | Limit requests to dependencies per instance |\n| **Bulkheads** | Isolate connection pools per service |\n| **Coordinated scaling** | Scale dependencies when app scales |\n| **Backpressure** | Reject requests when dependencies are slow |\n\n\n---\n\n# Auto Scaling Best Practices\n\n### Design Principles\n\n\n| Principle | Why | How |\n|-----------|-----|-----|\n| **Scale on leading indicators** | Respond before users are impacted | Use queue depth, request rate, not just CPU |\n| **Fail safe** | Over-capacity is safer than under | Conservative scale-in thresholds |\n| **Test at scale** | Validate scaling behavior | Regular load testing at 2x expected peak |\n| **Plan for scaling failures** | Scaling is not instant or guaranteed | Maintain buffer, implement degradation |\n| **Monitor scaling itself** | Catch configuration issues | Alert on frequent scaling, scaling failures |\n\n\n### Operational Guidelines\n\n\n| Guideline | Recommendation | Rationale |\n|-----------|----------------|-----------|\n| **Minimum capacity** | Never below baseline load handling | Survive if scaling fails |\n| **Maximum capacity** | Set explicit limit | Prevent runaway costs |\n| **Multiple metrics** | Scale on first to trigger | Catch different bottleneck types |\n| **Regular threshold review** | Adjust as patterns change | Traffic patterns evolve |\n| **Pre-event scaling** | Schedule for known events | Do not rely on reactive for launches |\n| **Health check accuracy** | Only healthy instances receive traffic | Bad instances should fail checks |\n\n\n### Cost Controls\n\n\n| Control | Purpose | Implementation |\n|---------|---------|----------------|\n| **Maximum instances** | Prevent runaway scaling | Set max in scaling group |\n| **Billing alerts** | Catch unexpected costs | Cloud billing alerts at thresholds |\n| **Scale-in policies** | Reduce capacity when not needed | Aggressive but safe thresholds |\n| **Spot/preemptible instances** | Lower cost for fault-tolerant | Mixed instance types |\n| **Reserved capacity** | Lower cost for baseline | Reserve minimum required |\n| **Right-sizing** | Optimal instance size | Regularly review instance types |\n\n\n---\n\n# Summary\n\nAuto-scaling automatically adjusts capacity to match demand, enabling efficient resource usage while maintaining reliability. Here are the key takeaways:\n\n\n| Concept | Key Point |\n|---------|-----------|\n| **Core loop** | Collect metrics → Compare to target → Decide → Execute → Stabilize → Repeat |\n| **Strategy selection** | Reactive for unknown patterns, scheduled for known patterns, predictive for regular patterns, combine all for comprehensive coverage |\n| **Metric selection** | Choose metrics that reflect actual bottlenecks. CPU works for compute-bound; use request count, queue depth, or latency for I/O-bound |\n| **Threshold design** | Asymmetric: scale out aggressively (70%), scale in conservatively (40%). Prevents oscillation. |\n| **Cooldowns** | Prevent thrashing. Short for scale-out (1-3 min), long for scale-in (5-10 min) |\n| **Environment differences** | VMs: 3-5 min, containers: 30s-2 min, serverless: instant but with cold starts |\n| **Common challenges** | Cold starts, connection exhaustion, thrashing, cascading failures. Plan for each. |\n\n\n#### **Key numbers to remember**\n\n\n| Setting | Typical Value | Notes |\n|---------|---------------|-------|\n| Target CPU utilization | 60-70% | Leave headroom for spikes |\n| Scale-out threshold | 70-80% | Trigger before overload |\n| Scale-in threshold | 30-40% | Wide gap prevents oscillation |\n| Scale-out cooldown | 1-3 minutes | Let new instances warm up |\n| Scale-in cooldown | 5-10 minutes | Ensure load is truly decreasing |\n| VM launch to traffic | 3-5 minutes | Plan for this lag |\n| Container launch to traffic | 30s-2 minutes | Much faster than VMs |\n| Minimum buffer capacity | 20-30% | Survive while scaling |\n\n\nAuto-scaling solves the capacity timing problem, but running more instances during peaks costs more than running a fixed small fleet. However, the waste of over-provisioning typically exceeds the cost of auto-scaling overhead. The next chapter explores cost optimization strategies that work alongside auto-scaling to minimize infrastructure spend while meeting performance requirements.\n\n---\n\n### References\n\n- [AWS Auto Scaling Documentation](undefined)\n- [Google Cloud Autoscaler](undefined)\n- [Kubernetes Horizontal Pod Autoscaler](undefined)\n- [KEDA - Kubernetes Event-Driven Autoscaling](undefined)\n- [Netflix Auto Scaling](undefined)\n- [AWS Well-Architected - Performance Efficiency](undefined)","pageType":"system-design"}

Auto Scaling

Ashish Pratap Singh

Get Premium