{"title":"Rolling Updates","description":"","content":"Imagine you run a Kubernetes cluster with 20 pods serving an e-commerce app and need to ship a bug fix fast. Blue-green would double your pods during the rollout, and canary needs traffic-splitting and metrics you might not have yet. \n\nWhat you want is simpler: update pods one at a time.\n\nThat is a **rolling update**. Kubernetes replaces instances incrementally, keeping the service available while old and new versions run side by side, without requiring duplicate infrastructure. When you apply a new image tag, you are usually triggering this default strategy.\n\nRolling updates are easy and cost-effective, but they trade control for simplicity. You do not get a true real-traffic test before committing (like canary), and rollback is not instant (like blue-green). For many stateless, backward-compatible services, the trade-off is worth it.\n\n---\n\n# The Mechanics of Rolling Updates\n\nA rolling update is conceptually simple: replace instances one at a time (or in small batches) until all instances run the new version. But the details matter. Let's trace through exactly what happens.\n\n### Step-by-Step Update Process\n\n\n```mermaid\nflowchart TB\n subgraph Step1[\"Step 1: Initial State\"]\n direction LR\n LB1[Load Balancer]:::orange\n A1[Pod A
v1.0]:::green\n B1[Pod B
v1.0]:::green\n C1[Pod C
v1.0]:::green\n D1[Pod D
v1.0]:::green\n LB1 --> A1\n LB1 --> B1\n LB1 --> C1\n LB1 --> D1\n end\n\n subgraph Step2[\"Step 2: First Pod Updating\"]\n direction LR\n LB2[Load Balancer]:::orange\n A2[Pod A
v2.0 starting]:::yellow\n B2[Pod B
v1.0]:::green\n C2[Pod C
v1.0]:::green\n D2[Pod D
v1.0]:::green\n LB2 --> B2\n LB2 --> C2\n LB2 --> D2\n end\n\n subgraph Step3[\"Step 3: First Pod Ready\"]\n direction LR\n LB3[Load Balancer]:::orange\n A3[Pod A
v2.0]:::primary\n B3[Pod B
v1.0]:::green\n C3[Pod C
v1.0]:::green\n D3[Pod D
v1.0]:::green\n LB3 --> A3\n LB3 --> B3\n LB3 --> C3\n LB3 --> D3\n end\n\n subgraph Step4[\"Step 4: Second Pod Updating\"]\n direction LR\n LB4[Load Balancer]:::orange\n A4[Pod A
v2.0]:::primary\n B4[Pod B
v2.0 starting]:::yellow\n C4[Pod C
v1.0]:::green\n D4[Pod D
v1.0]:::green\n LB4 --> A4\n LB4 --> C4\n LB4 --> D4\n end\n\n subgraph Step5[\"Step 5: Complete\"]\n direction LR\n LB5[Load Balancer]:::orange\n A5[Pod A
v2.0]:::primary\n B5[Pod B
v2.0]:::primary\n C5[Pod C
v2.0]:::primary\n D5[Pod D
v2.0]:::primary\n LB5 --> A5\n LB5 --> B5\n LB5 --> C5\n LB5 --> D5\n end\n\n Step1 --> Step2 --> Step3 --> Step4 --> Step5\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n```\n\n\n### The Pod Update Lifecycle\n\nEach pod goes through a precise sequence during replacement:\n\n\n```mermaid\nsequenceDiagram\n participant Orch as Orchestrator\n participant LB as Load Balancer\n participant OldPod as Old Pod (v1.0)\n participant NewPod as New Pod (v2.0)\n\n Note over OldPod: Running, serving traffic\n\n Note right of Orch: Phase 1: Drain Old Pod\n Orch->>LB: Remove OldPod from endpoints\n LB-->>OldPod: Stop sending new requests\n Note over OldPod: Finishing in-flight requests...\n Orch->>OldPod: Send SIGTERM\n OldPod->>OldPod: Graceful shutdown\n OldPod->>Orch: Exit\n\n Note right of Orch: Phase 2: Start New Pod\n Orch->>NewPod: Create container\n Note over NewPod: Starting up...\n NewPod->>NewPod: Initialize application\n\n Note right of Orch: Phase 3: Verify Health\n loop Until ready or timeout\n Orch->>NewPod: Readiness probe\n NewPod-->>Orch: Not ready yet\n\n NewPod->>NewPod: Finish initialization\n Orch->>NewPod: Readiness probe\n NewPod-->>Orch: Ready!\n end\n\n Note right of Orch: Phase 4: Serve Traffic\n Orch->>LB: Add NewPod to endpoints\n LB->>NewPod: Route traffic\n Note over NewPod: Running, serving traffic\n```\n\n\nThis sequence repeats for each pod until all pods run the new version. The orchestrator waits for each new pod to pass health checks before proceeding to the next, ensuring you never lose more capacity than configured.\n\n### Key Characteristics\n\n\n| Characteristic | Description | Implication |\n|----------------|-------------|-------------|\n| **Incremental** | Updates one pod at a time (or small batches) | Gradual risk exposure |\n| **In-place** | Reuses existing infrastructure | No duplicate environments needed |\n| **Mixed versions** | Both versions run simultaneously during update | Requires backward compatibility |\n| **Self-healing** | Failed pods trigger automatic recovery | Deployment can pause on errors |\n| **Resource efficient** | Only slightly more resources than steady state | Cost-effective for most deployments |\n\n\n---\n\n# Configuration Parameters\n\nTwo parameters control the behavior of rolling updates: `maxUnavailable` and `maxSurge`. Understanding these parameters is essential for balancing deployment speed, service availability, and resource usage.\n\n### Max Unavailable\n\nThis parameter limits how many pods can be unavailable during the update. \"Unavailable\" means the pod is either terminating, starting, or failing health checks.\n\n\n```mermaid\nflowchart LR\n\n subgraph Example2[\"maxUnavailable: 2 (from 4 replicas)\"]\n direction TB\n LB2[Load Balancer]:::orange\n A2[Pod A
UPDATING]:::yellow\n B2[Pod B
UPDATING]:::yellow\n C2[Pod C
v1.0]:::green\n D2[Pod D
v1.0]:::green\n Note2[Minimum 2 pods
always available]:::teal\n LB2 --> C2\n LB2 --> D2\n end\n\n subgraph Example1[\"maxUnavailable: 1 (from 4 replicas)\"]\n direction TB\n LB1[Load Balancer]:::orange\n A1[Pod A
UPDATING]:::yellow\n B1[Pod B
v1.0]:::green\n C1[Pod C
v1.0]:::green\n D1[Pod D
v1.0]:::green\n Note1[Minimum 3 pods
always available]:::teal\n LB1 --> B1\n LB1 --> C1\n LB1 --> D1\n end\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### **Configuration options**\n\n\n| Setting | Meaning | Use Case |\n|---------|---------|----------|\n| `maxUnavailable: 1` | At most 1 pod down at a time | Conservative, maximum availability |\n| `maxUnavailable: 25%` | At most 25% of desired pods down | Scales with deployment size |\n| `maxUnavailable: 0` | Never reduce below desired count | Zero capacity reduction (requires maxSurge > 0) |\n\n\n**Trade-off:** Higher values mean faster deployments but reduced capacity during the update. Lower values mean slower deployments but more available capacity.\n\n### Max Surge\n\nThis parameter limits how many extra pods can exist during the update. The orchestrator can create new pods before terminating old ones, temporarily exceeding the desired replica count.\n\n\n```mermaid\nflowchart LR\n\n subgraph Example2[\"maxSurge: 2 (4 replicas → 6 during update)\"]\n direction TB\n LB2[Load Balancer]:::orange\n A2[Pod A
v1.0]:::green\n B2[Pod B
v1.0]:::green\n C2[Pod C
v1.0]:::green\n D2[Pod D
v1.0]:::green\n E2[Pod E
v2.0 NEW]:::primary\n F2[Pod F
v2.0 NEW]:::primary\n Note2[6 total pods
during update]:::teal\n LB2 --> A2\n LB2 --> B2\n LB2 --> C2\n LB2 --> D2\n LB2 --> E2\n LB2 --> F2\n end\n\n subgraph Example1[\"maxSurge: 1 (4 replicas → 5 during update)\"]\n direction TB\n LB1[Load Balancer]:::orange\n A1[Pod A
v1.0]:::green\n B1[Pod B
v1.0]:::green\n C1[Pod C
v1.0]:::green\n D1[Pod D
v1.0]:::green\n E1[Pod E
v2.0 NEW]:::primary\n Note1[5 total pods
during update]:::teal\n LB1 --> A1\n LB1 --> B1\n LB1 --> C1\n LB1 --> D1\n LB1 --> E1\n end\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n**Configuration options:**\n\n\n| Setting | Meaning | Use Case |\n|---------|---------|----------|\n| `maxSurge: 1` | At most 1 extra pod during update | Minimal resource overhead |\n| `maxSurge: 25%` | At most 25% extra pods | Scales with deployment size |\n| `maxSurge: 0` | No extra pods allowed | Minimum resources (requires maxUnavailable > 0) |\n\n\n**Trade-off:** Higher values mean faster deployments and more capacity during updates, but higher resource usage. Lower values mean slower deployments but lower resource overhead.\n\n### Combining Parameters\n\nThe two parameters work together to control update behavior. Neither can be zero simultaneously (the update could not make progress).\n\n\n```mermaid\nflowchart TD\n subgraph Config1[\"Conservative: maxUnavailable=1, maxSurge=0\"]\n C1_Step[Replace one at a time
Always have N-1 running
Slowest, minimum resources]:::green\n end\n\n subgraph Config2[\"Zero-Downtime: maxUnavailable=0, maxSurge=1\"]\n C2_Step[Add new pod first
Then remove old pod
Always have N+ running]:::primary\n end\n\n subgraph Config3[\"Balanced: maxUnavailable=1, maxSurge=1\"]\n C3_Step[One starting, one stopping
simultaneously
Good balance of speed/safety]:::orange\n end\n\n subgraph Config4[\"Aggressive: maxUnavailable=25%, maxSurge=25%\"]\n C4_Step[Update up to 50% at once
Fastest, reduced capacity
Higher resource use]:::red\n end\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Practical example with 4 replicas:**\n\n\n| Configuration | Pods During Update | Speed | Resources |\n|---------------|-------------------|-------|-----------|\n| maxUnavailable=1, maxSurge=0 | 3-4 running | Slowest | Baseline |\n| maxUnavailable=0, maxSurge=1 | 4-5 running | Slow | +25% peak |\n| maxUnavailable=1, maxSurge=1 | 3-5 running | Medium | +25% peak |\n| maxUnavailable=2, maxSurge=2 | 2-6 running | Fast | +50% peak |\n\n\n---\n\n# Maintaining Zero-Downtime Availability\n\nA rolling update can achieve zero downtime, but only if several conditions are met. Miss any of these, and users will experience errors or dropped requests during deployments.\n\n### Health Checks: The Foundation\n\nHealth checks tell the orchestrator when a pod is ready to receive traffic. Without proper health checks, traffic might be routed to pods that are still initializing, causing errors.\n\n\n```mermaid\nsequenceDiagram\n participant Orch as Orchestrator\n participant Pod as New Pod\n participant LB as Load Balancer\n\n Orch->>Pod: Create container\n Note over Pod: Container starting...\n\n Note right of Pod: Application Initializing\n Pod->>Pod: Load configuration\n Pod->>Pod: Connect to database\n Pod->>Pod: Warm caches\n Pod->>Pod: Initialize thread pools\n\n loop Readiness Probes (every 5s)\n Orch->>Pod: GET /health/ready\n Pod-->>Orch: 503 Service Unavailable\n Note over Orch: Pod not ready, skip\n end\n\n Note over Pod: Initialization complete!\n\n Orch->>Pod: GET /health/ready\n Pod-->>Orch: 200 OK\n\n Orch->>LB: Add pod to endpoints\n Note over LB: Now routing traffic to pod\n\n LB->>Pod: User requests begin\n```\n\n\n### Liveness vs Readiness Probes\n\nThese two probe types serve different purposes and should not be configured identically:\n\n\n```mermaid\nflowchart TB\n subgraph Liveness[\"Liveness Probe\"]\n L_Q[Question: Is the process alive?]:::primary\n L_Pass[Pass: Container continues]:::green\n L_Fail[Fail: Container restarts]:::red\n L_Q --> L_Pass\n L_Q --> L_Fail\n end\n\n subgraph Readiness[\"Readiness Probe\"]\n R_Q[Question: Can it handle traffic?]:::primary\n R_Pass[Pass: Added to load balancer]:::green\n R_Fail[Fail: Removed from load balancer
Container keeps running]:::yellow\n R_Q --> R_Pass\n R_Q --> R_Fail\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n\n| Probe Type | Purpose | On Failure | Example Check |\n|------------|---------|------------|---------------|\n| **Liveness** | Is the process running and not deadlocked? | Restart container | Simple /health endpoint |\n| **Readiness** | Can the application serve requests? | Remove from load balancer | Database connected, caches warm |\n\n\n**Common mistakes:**\n\n- **No readiness probe:** Traffic hits pods before they are ready, causing errors\n- **Too aggressive liveness probe:** Kills slow-starting applications before they initialize\n- **Same check for both:** Liveness check should be simple; readiness can be comprehensive\n- **Too short initialDelaySeconds:** Probe fails before app has chance to start\n\n**Readiness probe example conditions:**\n\n- Database connection pool established\n- Required configuration loaded\n- Cache warm-up complete\n- Dependent services reachable\n- Application fully initialized\n\n### Graceful Shutdown\n\nWhen a pod is terminated, it must finish processing in-flight requests before exiting. Without graceful shutdown, users experience dropped connections and failed requests.\n\n\n```mermaid\nflowchart TD\n subgraph Shutdown[\"Graceful Shutdown Sequence\"]\n S1[SIGTERM received]:::orange\n S2[Stop accepting
new connections]:::orange\n S3[Finish in-flight
requests]:::yellow\n S4[Close database
connections]:::yellow\n S5[Flush logs
and metrics]:::yellow\n S6[Exit with code 0]:::green\n\n S1 --> S2 --> S3 --> S4 --> S5 --> S6\n end\n\n subgraph Timeout[\"Grace Period\"]\n T1[terminationGracePeriodSeconds
starts counting]:::teal\n T2[If exceeded:
SIGKILL sent]:::red\n end\n\n S1 --> T1\n T1 -.->|\"timeout\"| T2\n T2 --> Force[Forced termination
In-flight requests lost]:::red\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Timeline of pod termination:**\n\n\n| Time | Event | Note |\n|------|-------|------|\n| 0s | Pod marked for termination | Update begins |\n| 0s | Pod removed from Service endpoints | Concurrent with SIGTERM |\n| 0s | SIGTERM sent to container | Application should catch this |\n| 0-5s | Load balancer may still send requests | Race condition window |\n| 0-30s | Application completes in-flight requests | Graceful shutdown |\n| 30s | terminationGracePeriodSeconds expires | Default: 30 seconds |\n| 30s | SIGKILL sent | Forced termination |\n\n\n### The preStop Hook Solution\n\nThere is a race condition: the SIGTERM and endpoint removal happen simultaneously. Some load balancers may still route requests to the pod after SIGTERM is sent. A preStop hook solves this by delaying SIGTERM until endpoints have propagated:\n\n\n```mermaid\nsequenceDiagram\n participant Orch as Orchestrator\n participant LB as Load Balancer\n participant Pod as Pod\n\n Note over Pod: Running, serving traffic\n\n Orch->>Pod: Initiate termination\n Orch->>LB: Remove from endpoints\n\n Note right of Pod: preStop Hook Executes\n Pod->>Pod: sleep 5\n Note over LB: Endpoints update propagates...\n\n Note over LB: Pod removed from all load balancers\n\n Orch->>Pod: SIGTERM\n Pod->>Pod: Graceful shutdown\n Pod->>Pod: Complete in-flight requests\n Pod->>Orch: Exit\n```\n\n\n**preStop configuration:**\n\n\n```yaml\nlifecycle:\n preStop:\n exec:\n command: [\"sleep\", \"5\"]\n```\n\n\nThe 5-second sleep gives load balancers time to stop sending new requests before the application begins shutdown. This eliminates the race condition that causes dropped requests.\n\n**Interview Insight:** Understanding the termination race condition and preStop hook demonstrates deep operational knowledge. Many teams run into this issue in production without understanding why requests are dropped during deployments.\n\n---\n\n# The Challenge of Mixed Versions\n\nDuring a rolling update, some pods run the old version and some run the new version. This is fundamentally different from blue-green (instant switch) or canary (controlled traffic split). Users hit whichever version the load balancer routes them to, with no consistency guarantees.\n\n\n```mermaid\nflowchart LR\n Users([Users]):::rose\n LB[Load Balancer
Round-robin]:::orange\n\n LB --> V1A[v1.0]:::green\n LB --> V1B[v1.0]:::green\n LB --> V2A[v2.0]:::primary\n LB --> V2B[v2.0]:::primary\n\n subgraph UserExperience[\"User Experience\"]\n U1[Request 1 → v1.0]:::green\n U2[Request 2 → v2.0]:::primary\n U3[Request 3 → v1.0]:::green\n U4[Request 4 → v2.0]:::primary\n end\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n```\n\n\n### Problems This Creates\n\n**1. API Response Inconsistency**\n\n\n```mermaid\nflowchart TD\n subgraph Scenario[\"User Makes Two Requests\"]\n R1[Request 1
GET /user/123]:::rose\n R2[Request 2
GET /user/123]:::rose\n end\n\n subgraph Responses[\"Different Responses\"]\n Resp1[\"v1.0 Response:\n {\n name: 'John',\n email: 'john@example.com'\n }\"]:::green\n\n Resp2[\"v2.0 Response:\n {\n name: 'John',\n email: 'john@example.com',\n phone: '+1-555-0123' ← NEW\n }\"]:::primary\n end\n\n R1 --> Resp1\n R2 --> Resp2\n\n Problem[Client may be confused
by inconsistent schema]:::yellow\n\n Responses --> Problem\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n```\n\n\n**2. Feature Availability Inconsistency**\n\nA user sees a new button in the UI (served by v2.0), clicks it, but the request lands on v1.0 which does not have the endpoint. Result: 404 error.\n\n**3. Session State Mismatch**\n\nUser logs into v1.0, session stored in memory. Next request goes to v2.0 with different session format. User appears logged out or session is corrupted.\n\n**4. Database Schema Conflicts**\n\nv2.0 writes data in a new format. v1.0 pod reads it and fails to parse. Or worse, v1.0 overwrites it with the old format, losing data.\n\n### Solutions for Mixed Version Traffic\n\n**Backward-Compatible API Changes**\n\nDesign changes so that old clients can work with new responses and new clients can work with old responses:\n\n\n| Change Type | Backward Compatible? | Safe for Rolling Update? |\n|-------------|---------------------|-------------------------|\n| Add optional field | Yes | Yes |\n| Add new endpoint | Yes | Yes |\n| Make required field optional | Yes | Yes |\n| Remove field | No | No - needs versioning |\n| Rename field | No | No - needs versioning |\n| Change field type | No | No - needs versioning |\n| Remove endpoint | No | No - needs versioning |\n\n\n**Sticky Sessions**\n\nRoute the same user to the same version throughout their session:\n\n\n```mermaid\nflowchart LR\n UserA([User A]):::rose\n UserB([User B]):::rose\n\n LB[Load Balancer
Session affinity]:::orange\n\n LB -->|\"Cookie: server=v1-pod-a\"| V1[v1.0]:::green\n LB -->|\"Cookie: server=v2-pod-b\"| V2[v2.0]:::primary\n\n UserA -->|\"All requests\"| V1\n UserB -->|\"All requests\"| V2\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n```\n\n\nThis reduces inconsistency but does not eliminate it. When the user's v1.0 pod is updated, they will be routed to a v2.0 pod for subsequent requests.\n\n**Feature Flags**\n\nDecouple deployment from feature release. Deploy code that checks a feature flag before exposing new functionality:\n\n\n```mermaid\nflowchart TD\n Request[User Request]:::rose\n Flag{Feature flag
enabled?}:::orange\n\n NewFeature[Show new feature]:::primary\n OldBehavior[Show old behavior]:::green\n\n Request --> Flag\n Flag -->|\"Yes\"| NewFeature\n Flag -->|\"No\"| OldBehavior\n\n Note1[Both v1.0 and v2.0 can
have the same behavior
until flag is enabled]:::teal\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nThis lets you deploy code without activating features. Once all pods are updated, flip the flag to enable the feature for everyone simultaneously.\n\n---\n\n# Rollback Strategies\n\nWhen a rolling update goes wrong, you need to return to a working state. Unlike blue-green deployments where rollback is instant, rolling update rollback is itself a rolling update, just in reverse.\n\n### How Rollback Works\n\n\n```mermaid\nflowchart TD\n subgraph Initial[\"Rolling Update Failed\"]\n State1[2 pods v2.0 FAILING
2 pods v1.0 HEALTHY]:::yellow\n Note1[Update paused due
to health check failures]:::orange\n end\n\n subgraph Rollback[\"Rollback Command Issued\"]\n State2[Rolling update to v1.0
begins]:::primary\n end\n\n subgraph Progress[\"Rollback In Progress\"]\n State3[Replacing v2.0 pods
with v1.0 pods]:::primary\n end\n\n subgraph Complete[\"Rollback Complete\"]\n State4[4 pods v1.0
All healthy]:::green\n end\n\n Initial --> Rollback --> Progress --> Complete\n\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n### Automatic vs Manual Rollback\n\n**Automatic Pause on Failure**\n\nThe orchestrator automatically pauses the rollout when new pods fail health checks:\n\n\n```mermaid\nflowchart TD\n Update[Start Rolling Update]:::green\n NewPod[Create New Pod]:::green\n Health{Health Check
Pass?}:::orange\n\n Success[Pod Ready
Continue to next]:::green\n Retry[Retry Pod
Creation]:::yellow\n RetryHealth{Health Check
Pass?}:::orange\n Pause[Pause Rollout]:::red\n Mixed[System in mixed state
Manual intervention needed]:::red\n\n Update --> NewPod --> Health\n Health -->|\"Yes\"| Success\n Health -->|\"No\"| Retry\n Retry --> RetryHealth\n RetryHealth -->|\"Yes\"| Success\n RetryHealth -->|\"No\"| Pause\n Pause --> Mixed\n\n Success --> |\"More pods?\"| NewPod\n Success --> |\"All done\"| Complete[Update Complete]:::green\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nWhen paused, the system is in a mixed state. You must investigate and then either:\n\n1. Fix the issue and resume the rollout\n2. Roll back to the previous version\n\n**Manual Rollback Commands**\n\n\n```shell\n# View rollout status\nkubectl rollout status deployment/my-app\n\n# View revision history\nkubectl rollout history deployment/my-app\n\n# Roll back to previous version\nkubectl rollout undo deployment/my-app\n\n# Roll back to specific revision\nkubectl rollout undo deployment/my-app --to-revision=2\n\n# Pause an ongoing rollout\nkubectl rollout pause deployment/my-app\n\n# Resume a paused rollout\nkubectl rollout resume deployment/my-app\n```\n\n\n### Rollback Speed Comparison\n\n\n| Strategy | Rollback Mechanism | Time to Recover |\n|----------|-------------------|-----------------|\n| **Blue-Green** | Switch traffic to old environment | Seconds |\n| **Canary** | Remove canary from rotation | Seconds |\n| **Rolling Update** | Rolling update in reverse | Minutes (same as original update) |\n\n\nRolling update rollback is slower because you go through the same gradual replacement process. During rollback, you again have mixed versions as v2.0 pods are replaced with v1.0 pods.\n\n### Minimizing Rollback Impact\n\n**Keep Previous Revision Available**\n\nKubernetes keeps revision history by default. Ensure `revisionHistoryLimit` is not set too low:\n\n\n```yaml\nspec:\n revisionHistoryLimit: 10 # Keep 10 previous revisions\n```\n\n\n**Use progressDeadlineSeconds**\n\nSet a deadline for the rollout to complete. If exceeded, the rollout is marked as failed:\n\n\n```yaml\nspec:\n progressDeadlineSeconds: 600 # 10 minutes\n```\n\n\nThis prevents indefinite partial updates. If the update cannot complete in the deadline, it is automatically marked as failed, making it clear that action is needed.\n\n**Monitor During Rollout**\n\nWatch key metrics during deployment:\n\n- Error rate (should not increase)\n- Latency (should not spike significantly)\n- Pod restart count (should be 0)\n- Rollout progress (pods transitioning as expected)\n\n---\n\n# Kubernetes Configuration Deep Dive\n\nKubernetes Deployments are the most common way to implement rolling updates. Let's examine a production-ready configuration.\n\n### Complete Deployment Spec\n\n\n```yaml\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: my-app\n labels:\n app: my-app\nspec:\n replicas: 4\n revisionHistoryLimit: 10\n progressDeadlineSeconds: 600\n\n strategy:\n type: RollingUpdate\n rollingUpdate:\n maxUnavailable: 1\n maxSurge: 1\n\n selector:\n matchLabels:\n app: my-app\n\n template:\n metadata:\n labels:\n app: my-app\n version: v2.0\n spec:\n terminationGracePeriodSeconds: 30\n\n containers:\n - name: app\n image: my-app:v2.0\n\n ports:\n - containerPort: 8080\n\n resources:\n requests:\n cpu: 100m\n memory: 256Mi\n limits:\n cpu: 500m\n memory: 512Mi\n\n readinessProbe:\n httpGet:\n path: /health/ready\n port: 8080\n initialDelaySeconds: 5\n periodSeconds: 5\n timeoutSeconds: 3\n failureThreshold: 3\n\n livenessProbe:\n httpGet:\n path: /health/live\n port: 8080\n initialDelaySeconds: 15\n periodSeconds: 10\n timeoutSeconds: 3\n failureThreshold: 3\n\n lifecycle:\n preStop:\n exec:\n command: [\"sleep\", \"5\"]\n```\n\n\n### Key Configuration Explained\n\n\n| Setting | Purpose | Recommendation |\n|---------|---------|----------------|\n| `replicas` | Desired pod count | Based on load requirements |\n| `revisionHistoryLimit` | Revisions to keep for rollback | 10 (default), higher for frequent deployments |\n| `progressDeadlineSeconds` | Max time for rollout | 2-3x expected rollout duration |\n| `maxUnavailable` | Max pods down during update | 25% or 1 for balanced approach |\n| `maxSurge` | Max extra pods during update | 25% or 1 for balanced approach |\n| `terminationGracePeriodSeconds` | Time for graceful shutdown | Longest expected request + buffer |\n| `initialDelaySeconds` | Wait before first probe | Time for app to start |\n| `periodSeconds` | Probe interval | 5-10 seconds |\n| `failureThreshold` | Failed probes before action | 3 for stability |\n\n\n### Pod Disruption Budgets\n\nPod Disruption Budgets (PDBs) provide an additional layer of protection, ensuring minimum availability across all disruptions, not just rolling updates:\n\n\n```yaml\napiVersion: policy/v1\nkind: PodDisruptionBudget\nmetadata:\n name: my-app-pdb\nspec:\n minAvailable: 3 # Always keep at least 3 pods running\n selector:\n matchLabels:\n app: my-app\n```\n\n\nAlternative configuration:\n\n\n```yaml\nspec:\n maxUnavailable: 1 # Never have more than 1 pod unavailable\n```\n\n\n**PDBs protect against:**\n\n- Rolling updates removing too many pods\n- Node drains (maintenance, scaling)\n- Cluster autoscaler removing nodes\n- Manual pod deletions\n- Any voluntary disruption\n\n\n```mermaid\nflowchart TD\n subgraph Disruptions[\"Voluntary Disruptions\"]\n\t\tdirection TB\n D1[Rolling Update]:::orange\n D2[Node Drain]:::orange\n D3[Cluster Scaling]:::orange\n D4[Manual Delete]:::orange\n end\n\n PDB[Pod Disruption Budget
minAvailable: 3]:::primary\n\n subgraph Protection[\"PDB Enforcement\"]\n Allow[Allow disruption
if >= 3 pods remain]:::green\n Block[Block disruption
if < 3 pods would remain]:::red\n end\n\n Disruptions --> PDB\n PDB --> Allow\n PDB --> Block\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n---\n\n# When to Choose Rolling Updates\n\nRolling updates are the default for good reason, but they are not always the best choice.\n\n### Ideal Scenarios\n\n\n| Scenario | Why Rolling Updates Work |\n|----------|-------------------------|\n| **Stateless applications** | No session or state synchronization needed |\n| **Backward-compatible changes** | Mixed versions work correctly together |\n| **Cost-sensitive environments** | No duplicate infrastructure required |\n| **Simple validation** | Health checks sufficient to verify readiness |\n| **Frequent deployments** | Low overhead for regular releases |\n| **Kubernetes-native apps** | Built-in support, familiar tooling |\n\n\n### Poor Fit Scenarios\n\n\n| Scenario | Why Rolling Updates Struggle |\n|----------|------------------------------|\n| **Breaking API changes** | Mixed versions cause compatibility errors |\n| **Instant rollback required** | Rollback is another rolling update |\n| **Complex pre-production testing** | Cannot test with real traffic before commit |\n| **Stateful applications** | State synchronization during mixed versions is complex |\n| **Database schema changes** | Requires careful expand-contract migrations |\n\n\n### Comparison with Other Strategies\n\n\n```mermaid\nflowchart TD\n Question{What matters most?}:::orange\n\n Question -->|\"Instant rollback\"| BG[Blue-Green]:::primary\n Question -->|\"Test with real traffic\"| Canary[Canary]:::green\n Question -->|\"Simplicity & cost\"| Rolling[Rolling Update]:::teal\n\n BG --> BG_Use[Critical systems
Infrequent releases
Need instant recovery]:::primary\n Canary --> Canary_Use[High traffic volume
Subtle bugs likely
Good observability]:::green\n Rolling --> Rolling_Use[Stateless services
Backward-compatible
Frequent releases]:::teal\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n\n| Factor | Blue-Green | Canary | Rolling |\n|--------|------------|--------|---------|\n| **Rollback Speed** | Instant | Fast | Slow |\n| **Resource Overhead** | 2x during deploy | ~1.1x | ~1.1x |\n| **Complexity** | Medium | High | Low |\n| **Pre-deploy Testing** | Full environment | Partial traffic | Health checks only |\n| **Mixed Versions** | No | Intentional, controlled | Incidental, uncontrolled |\n| **Blast Radius** | 100% or 0% | Configurable % | Varies during update |\n\n\n---\n\n# Best Practices Summary\n\n### Health Check Configuration\n\n**Do:**\n\n- Use separate endpoints for liveness and readiness\n- Set `initialDelaySeconds` based on actual startup time\n- Make readiness check comprehensive (database, dependencies)\n- Make liveness check simple (process alive, not deadlocked)\n\n**Avoid:**\n\n- Same check for liveness and readiness\n- Too aggressive `periodSeconds` or `failureThreshold`\n- Health checks that depend on external services (for liveness)\n- Missing readiness probes\n\n### Graceful Shutdown\n\n**Do:**\n\n- Handle SIGTERM in your application\n- Stop accepting new connections immediately\n- Complete in-flight requests before exiting\n- Use preStop hook to handle race conditions\n- Set `terminationGracePeriodSeconds` appropriately\n\n**Avoid:**\n\n- Ignoring SIGTERM\n- Immediate exit on shutdown\n- `terminationGracePeriodSeconds` shorter than max request time\n\n### Update Configuration\n\n**Do:**\n\n- Set both `maxUnavailable` and `maxSurge` explicitly\n- Use Pod Disruption Budgets\n- Set `progressDeadlineSeconds`\n- Monitor during rollouts\n- Keep revision history for rollback\n\n**Avoid:**\n\n- Default configuration without consideration\n- `maxUnavailable: 0` with `maxSurge: 0`\n- Unlimited revision history (storage)\n- Ignoring rollout status\n\n---\n\n# Summary\n\nRolling updates provide a simple, resource-efficient deployment strategy that works well for most applications:\n\n**Core mechanism:** Replace pods one at a time, ensuring health checks pass before proceeding to the next pod. Service availability is maintained throughout.\n\n**Key parameters:**\n\n- `maxUnavailable` controls minimum available capacity\n- `maxSurge` controls maximum resource usage\n- Balance these based on availability requirements and resource constraints\n\n**Zero-downtime requirements:**\n\n- Readiness probes must accurately reflect ability to serve traffic\n- Graceful shutdown must complete in-flight requests\n- preStop hooks handle the termination race condition\n- terminationGracePeriodSeconds must exceed longest request\n\n**Mixed version challenges:** During updates, both versions run simultaneously. This requires backward-compatible changes, externalized session state, and careful database schema management.\n\n**Rollback limitations:** Unlike blue-green, rollback is another rolling update, taking minutes rather than seconds. Monitor during rollouts and keep revision history for quick rollback commands.\n\n**Best for:** Stateless services with backward-compatible changes, cost-sensitive environments, and teams wanting simple, well-understood deployment mechanics.\n\nRolling updates give you incremental control over infrastructure changes, but they do not give you control over which users see which features. What if you want to enable new functionality for specific users without redeploying, or test different implementations in production? This is where feature flags come in, allowing you to control feature availability at runtime independent of deployment.\n\n---\n\n### References\n\n- [Kubernetes Deployments](undefined)\n- [Kubernetes Rolling Updates Tutorial](undefined)\n- [Container Probes](undefined)\n- [Pod Disruption Budgets](undefined)\n- [Graceful Shutdown in Kubernetes](undefined)\n- [Zero Downtime Deployment with Kubernetes](undefined)\n- [Kubernetes Deployment Strategies](undefined)","pageType":"system-design"}

Rolling Updates

Ashish Pratap Singh

Get Premium