{"title":"Feature Flags","description":"","content":"Imagine your team has spent three months building a new checkout flow that touches payments, inventory, and order confirmation. Shipping it all at once is risky. If it breaks, you are looking at an emergency rollback, angry customers, and a ruined weekend.\n\nFeature flags take a safer route. You deploy the code with the new checkout off, then enable it gradually: internal users first, then 5%, 25%, 50%, and finally 100%, watching metrics at each step. If anything goes wrong, you flip the flag off in seconds without rolling back the deployment.\n\nThat is the core idea: feature flags decouple deployment from release. Code can be in production while the feature stays hidden until you choose to expose it. With that separation, you get controlled rollouts, quick kill switches, A/B tests, and smoother trunk-based development.\n\nThey are powerful, but they add complexity. Knowing how to manage flags and when to remove them is part of operating mature production systems.\n\n---\n\n# The Problem Feature Flags Solve\n\nEvery deployment strategy we have discussed so far, blue-green, canary, rolling updates, controls which version of code runs in production. To change behavior, you deploy new code. Feature flags flip this model: deploy the code once, then control its behavior through configuration at runtime.\n\n\n```mermaid\nflowchart LR\n\n subgraph FeatureFlags[\"With Feature Flags\"]\n\t\tdirection LR\n F1[Develop Feature]:::green\n F2[Test in Staging]:::green\n F3[Deploy with Flag OFF]:::green\n F4[Enable for Internal Team]:::green\n F5[Enable for 5% of Users]:::orange\n F6[Problem Discovered]:::red\n F7[Disable Flag Instantly
No Deployment]:::green\n\n F1 --> F2 --> F3 --> F4 --> F5 --> F6 --> F7\n end\n\n subgraph Traditional[\"Traditional Release\"]\n\t\tdirection LR\n T1[Develop Feature]:::green\n T2[Test in Staging]:::green\n T3[Deploy to Production]:::orange\n T4[Feature Live for ALL Users]:::primary\n T5[Problem Discovered]:::red\n T6[Emergency Rollback
Deploy Previous Version]:::red\n\n T1 --> T2 --> T3 --> T4 --> T5 --> T6\n end\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe difference in recovery time is dramatic. Rolling back a deployment might take 10-30 minutes depending on your infrastructure. Disabling a feature flag takes seconds. During an incident, that difference matters enormously.\n\n### Benefits of Feature Flags\n\n\n| Benefit | Description | Example |\n|---------|-------------|---------|\n| **Instant rollback** | Disable a flag in seconds, no deployment needed | Checkout bug discovered, flag off in 5 seconds |\n| **Gradual rollout** | Enable for 1%, then 10%, then 50%, then 100% | New search algorithm rolled out over a week |\n| **User targeting** | Enable for specific users, groups, or segments | Internal team tests in production |\n| **Trunk-based development** | Merge incomplete features safely behind flags | Half-done feature merged to main, hidden from users |\n| **Operational control** | Disable expensive features during incidents | Turn off recommendations when database is overloaded |\n| **Testing in production** | Test with real data without affecting users | Dark launch validates new payment processor |\n\n\n### The Deployment vs Release Distinction\n\nThis distinction is worth emphasizing because it changes how you think about shipping software:\n\n\n```mermaid\nflowchart LR\n subgraph Deployment[\"Deployment (Code in Production)\"]\n D1[Monday: Deploy v2.0
Flag: OFF]:::primary\n D2[Wednesday: Deploy v2.1
Bug fix, Flag: OFF]:::primary\n end\n\n subgraph Release[\"Release (Feature Available)\"]\n R1[Tuesday: Enable for
internal users]:::green\n R2[Thursday: Enable for
10% of users]:::green\n R3[Friday: Enable for
100% of users]:::green\n end\n\n D1 --> R1\n D2 --> R2\n R2 --> R3\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nYou can deploy code on Monday and release the feature on Friday. You can deploy a bug fix on Wednesday without changing who has access to the feature. You can even deploy the code and never release it if you decide the feature is not ready.\n\n---\n\n# Types of Feature Flags\n\nNot all feature flags serve the same purpose. Using the wrong type leads to confusion about how long a flag should live and who should control it. Understanding the types helps you choose the right approach.\n\n### Release Flags\n\nRelease flags control the rollout of new features. They are the most common type and what most people think of when they hear \"feature flags.\"\n\n\n```mermaid\nflowchart LR\n Request([User Request]):::rose\n Check{new_checkout
enabled?}:::orange\n\n Check -->|\"Yes\"| New[New Checkout Flow]:::green\n Check -->|\"No\"| Old[Old Checkout Flow]:::primary\n\n Request --> Check\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n```\n\n\n**Characteristics:**\n\n- Short-lived: days to weeks\n- Binary (on/off) or percentage-based\n- Should be removed after full rollout\n- Owned by the feature team\n\n**Lifecycle:** Create when development starts, use during rollout, remove within two weeks of reaching 100%.\n\n### Experiment Flags\n\nExperiment flags enable A/B testing and multivariate experiments. They differ from release flags because they are designed to measure which variant performs better, not just to control access.\n\n\n```mermaid\nflowchart TD\n Request([User Request]):::rose\n Hash[Hash user ID to
assign variant]:::orange\n\n Hash -->|\"0-33\"| A[Variant A
Blue Button]:::primary\n Hash -->|\"34-66\"| B[Variant B
Green Button]:::green\n Hash -->|\"67-99\"| C[Control
Original Gray]:::teal\n\n A --> Track[Track Metrics
Click rate, Conversion]:::purple\n B --> Track\n C --> Track\n\n Track --> Analyze[Statistical Analysis
Which wins?]:::purple\n\n Request --> Hash\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n**Characteristics:**\n\n- Medium-lived: weeks to months\n- Multiple variants, not just on/off\n- Requires statistical rigor to declare a winner\n- Owned by product/growth teams\n- Integrated with analytics\n\n**Lifecycle:** Create when experiment starts, run until statistically significant results, implement winner and remove flag.\n\n### Ops Flags (Kill Switches)\n\nOps flags control system behavior during incidents or high load. They are manual circuit breakers that let you quickly disable expensive or problematic functionality.\n\n\n```mermaid\nflowchart TD\n Request([User Request]):::rose\n Check{recommendations
enabled?}:::orange\n\n Check -->|\"Yes\"| Call[Call Recommendation
Service]:::green\n Call --> Timeout{Response
in time?}\n Timeout -->|\"Yes\"| Return[Return
Personalized Results]:::green\n Timeout -->|\"No\"| Fallback\n\n Check -->|\"No\"| Fallback[Return Static
Default Recommendations]:::teal\n\n Request --> Check\n\n Note[During incident:
Disable flag instantly
to reduce load]:::yellow\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n```\n\n\n#### **Characteristics**\n\n- Long-lived: permanent\n- Binary (on/off), usually defaults to ON\n- Never removed (they document operational dependencies)\n- Owned by the on-call team\n- Should be exercised periodically to ensure they work\n\n**Lifecycle:** Create when the feature is built, keep forever, periodically verify the fallback path works.\n\n### Permission Flags\n\nPermission flags control access to features based on user attributes like subscription tier, role, or entitlements. They are not really about releases; they are about product capabilities.\n\n\n```mermaid\nflowchart TD\n User([User Request]):::rose\n Check{Check User
Entitlements}:::orange\n\n Check -->|\"Enterprise Plan\"| Ent[Enterprise Features
SSO, Audit Logs, API]:::purple\n Check -->|\"Pro Plan\"| Pro[Pro Features
Advanced Analytics]:::primary\n Check -->|\"Free Plan\"| Free[Free Features
Basic Functionality]:::green\n\n User --> Check\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n**Characteristics:**\n\n- Long-lived: tied to product/pricing model\n- Based on user attributes or entitlements\n- Often integrated with billing systems\n- Owned by product/billing teams\n- Different from release flags (these are about access, not rollout)\n\n**Lifecycle:** Create when the pricing tier is defined, modify when pricing changes, remove only when the product model changes.\n\n### Flag Type Summary\n\n\n| Type | Lifespan | Purpose | Owner | Example |\n|------|----------|---------|-------|---------|\n| **Release** | Days-weeks | Control rollout | Feature team | new_checkout |\n| **Experiment** | Weeks-months | Measure variants | Growth team | button_color_test |\n| **Ops** | Permanent | Operational control | On-call team | enable_recommendations |\n| **Permission** | Long-term | Access control | Product team | enterprise_sso |\n\n\n---\n\n# Implementation Architecture\n\nA feature flag system ranges from a simple configuration file to a sophisticated distributed platform. The right choice depends on your scale, requirements, and team size.\n\n### Simple Implementation: Configuration File\n\nThe simplest approach is a configuration file or environment variable:\n\n\n```mermaid\nflowchart TB\n subgraph App[\"Application\"]\n Code[Application Code]:::primary\n Config[Config File
flags.json]:::orange\n end\n\n Code -->|\"Read on startup\"| Config\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n\n```json\n{\n \"flags\": {\n \"new_checkout\": false,\n \"dark_mode\": true,\n \"api_v2\": false\n }\n}\n```\n\n\n**Pros:** Simple, no external dependencies, works offline **Cons:** Requires deployment to change flags, no targeting, no gradual rollout\n\nThis works for small teams or simple use cases but does not scale.\n\n### Production Architecture\n\nA production feature flag system has multiple components working together:\n\n\n```mermaid\nflowchart TB\n subgraph Applications[\"Applications\"]\n Web[Web App]:::primary\n Mobile[Mobile App]:::primary\n API[Backend API]:::primary\n end\n\n subgraph SDKs[\"Feature Flag SDKs\"]\n WebSDK[JS SDK]:::orange\n MobileSDK[Mobile SDK]:::orange\n ServerSDK[Server SDK]:::orange\n end\n\n subgraph Cache[\"Local Caching\"]\n WebCache[In-Memory Cache]:::green\n MobileCache[Local Storage]:::green\n ServerCache[Redis/Memory]:::green\n end\n\n subgraph FlagService[\"Flag Management Service\"]\n API_GW[Flag API]:::purple\n Stream[Streaming Server
SSE/WebSocket]:::purple\n Admin[Admin Dashboard]:::purple\n Store[(Flag Store
PostgreSQL)]:::purple\n end\n\n Web --> WebSDK --> WebCache\n Mobile --> MobileSDK --> MobileCache\n API --> ServerSDK --> ServerCache\n\n WebCache <-->|\"Fetch/Stream\"| API_GW\n MobileCache <-->|\"Fetch/Stream\"| API_GW\n ServerCache <-->|\"Fetch/Stream\"| API_GW\n\n API_GW <--> Store\n Stream <--> Store\n Admin --> API_GW\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n**Components:**\n\n\n| Component | Purpose |\n|-----------|---------|\n| **SDK** | Provides simple API for checking flags in application code |\n| **Local Cache** | Stores flag rules locally for fast evaluation |\n| **Flag API** | Central service that manages flag definitions and rules |\n| **Streaming Server** | Pushes flag changes to SDKs in real-time |\n| **Admin Dashboard** | UI for creating, modifying, and monitoring flags |\n| **Flag Store** | Persistent storage for flag configurations |\n\n\n### SDK Integration\n\nThe SDK provides a clean API for checking flags:\n\n\n```python\n# Initialize SDK with API key\nflags = FeatureFlagClient(api_key=\"sdk-xxx\")\n\n# Simple boolean check\nif flags.is_enabled(\"new_checkout\"):\n show_new_checkout()\nelse:\n show_old_checkout()\n\n# Check with user context for targeting\nuser = {\"id\": \"user-123\", \"email\": \"alice@company.com\", \"plan\": \"premium\"}\nif flags.is_enabled(\"premium_feature\", user):\n show_premium_feature()\n\n# Get variant for A/B test\nvariant = flags.get_variant(\"button_color_test\", user)\nrender_button(color=variant) # Returns \"blue\", \"green\", or \"control\"\n\n# Get configuration value\ntimeout = flags.get_value(\"api_timeout_ms\", default=5000)\n```\n\n\n### Evaluation Strategies\n\nFlags can be evaluated in different locations, each with trade-offs:\n\n\n```mermaid\nflowchart TB\n subgraph ServerSide[\"Server-Side Evaluation\"]\n S1[Full user context available]:::green\n S2[Flag logic hidden from clients]:::green\n S3[Secure, rules not exposed]:::green\n S4[Latency: network call or cache lookup]:::yellow\n end\n\n subgraph ClientSide[\"Client-Side Evaluation\"]\n C1[Instant evaluation, no network]:::green\n C2[Works offline]:::green\n C3[Flag rules visible in client code]:::red\n C4[Limited context available]:::yellow\n end\n\n subgraph Edge[\"Edge Evaluation (CDN)\"]\n E1[Low latency globally]:::green\n E2[Reduces origin load]:::green\n E3[Limited to simple rules]:::yellow\n E4[Cache invalidation complexity]:::yellow\n end\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n\n| Strategy | Latency | Security | Use Case |\n|----------|---------|----------|----------|\n| **Server-side** | Low (cached) | High | Backend services, sensitive rules |\n| **Client-side** | Lowest | Low | UI toggles, non-sensitive features |\n| **Edge** | Very low | Medium | Global applications, simple targeting |\n\n\nMost production systems use server-side evaluation with local caching: the SDK downloads all flag rules on startup and evaluates them locally without network calls.\n\n---\n\n# Targeting and Segmentation\n\nThe real power of feature flags comes from controlling who sees what. Simple on/off toggles are useful, but targeting specific users, segments, or percentages enables sophisticated rollout strategies.\n\n### Individual User Targeting\n\nEnable features for specific users by ID or email:\n\n\n```mermaid\nflowchart TD\n User([User Request]):::rose\n Rules{Evaluate Rules}:::orange\n\n Rules -->|\"user.email = 'alice@company.com'\"| Yes1[Feature Enabled]:::green\n Rules -->|\"user.email = 'bob@company.com'\"| Yes2[Feature Enabled]:::green\n Rules -->|\"Otherwise\"| No[Feature Disabled]:::primary\n\n User --> Rules\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n```\n\n\n**Use cases:**\n\n- Internal testing: enable for @company.com emails\n- Customer access: enable for specific customer IDs\n- Debugging: enable for a user reporting issues\n\n### Attribute-Based Targeting\n\nEnable based on user properties or request context:\n\n\n```mermaid\nflowchart TD\n User([User Request]):::rose\n Rules{Evaluate Targeting Rules}:::orange\n\n Rules -->|\"user.country = 'US'\"| US[US-specific Experience]:::primary\n Rules -->|\"user.plan = 'enterprise'\"| Ent[Enterprise Features]:::purple\n Rules -->|\"user.createdAt < 30 days ago\"| New[New User Onboarding]:::green\n Rules -->|\"request.platform = 'ios'\"| iOS[iOS-specific Feature]:::teal\n Rules -->|\"Default\"| Default[Standard Experience]:::teal\n\n User --> Rules\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n**Common targeting attributes:**\n\n\n| Category | Attributes |\n|----------|------------|\n| **User** | id, email, plan, role, createdAt, lastLoginAt |\n| **Geography** | country, region, city, timezone |\n| **Platform** | device, os, browser, appVersion |\n| **Account** | companyId, companySize, industry |\n| **Computed** | isInternal, isBetaTester, cohort |\n\n\n### Percentage Rollouts\n\nEnable for a percentage of users using consistent hashing:\n\n\n```mermaid\nflowchart LR\n\n subgraph Example3[\"Third User\"]\n\t\tdirection LR\n User3([\"user_id: def456\"]):::rose\n Hash3[\"hash(def456) = 23419\"]:::orange\n Mod3[\"23419 mod 100 = 19\"]:::orange\n Check3{\"19 < 25?\"}:::yellow\n\n Check3 -->|\"Yes (19 < 25)\"| Enabled3[\"Feature Enabled\"]:::green\n end\n\t\n subgraph Example2[\"Another User\"]\n\t\tdirection LR\n User2([\"user_id: xyz789\"]):::rose\n Hash2[\"hash(xyz789) = 142857\"]:::orange\n Mod2[\"142857 mod 100 = 57\"]:::orange\n Check2{\"57 < 25?\"}:::yellow\n\n Check2 -->|\"No (57 >= 25)\"| Disabled2[\"Feature Disabled\"]:::primary\n end\t\n\n subgraph Process[\"Percentage Rollout: 25%\"]\n\t\tdirection LR\n User([\"user_id: abc123\"]):::rose\n Hash[\"hash(abc123) = 847291\"]:::orange\n Mod[\"847291 mod 100 = 91\"]:::orange\n Check{\"91 < 25?\"}:::yellow\n\n Check -->|\"No (91 >= 25)\"| Disabled[\"Feature Disabled\"]:::primary\n end\n\n User --> Hash --> Mod --> Check\n User2 --> Hash2 --> Mod2 --> Check2\n User3 --> Hash3 --> Mod3 --> Check3\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### **Why consistent hashing matters**\n\nThe same user always gets the same result. Without consistent hashing, a user might see a feature on one request and not on the next, which is confusing and can break functionality. With consistent hashing:\n\n- user_id \"abc123\" always hashes to 91, always disabled at 25%\n- Increase rollout to 50%: 91 < 50? No, still disabled\n- Increase rollout to 95%: 91 < 95? Yes, now enabled\n- User never sees the feature appear and disappear randomly\n\n### Rule Ordering and Precedence\n\nWhen multiple rules apply, order matters. Rules are evaluated top to bottom, first match wins:\n\n\n```mermaid\nflowchart TD\n User([User Request]):::rose\n\n R1{Rule 1:
user.isBanned?}:::red\n R2{Rule 2:
user.isInternal?}:::purple\n R3{Rule 3:
user.plan = 'enterprise'?}:::primary\n R4{Rule 4:
Percentage: 25%}:::orange\n R5[Default: OFF]:::teal\n\n R1 -->|\"Yes\"| OFF1[Return FALSE
Banned users never get features]:::red\n R1 -->|\"No\"| R2\n\n R2 -->|\"Yes\"| ON1[Return TRUE
Internal users always get features]:::green\n R2 -->|\"No\"| R3\n\n R3 -->|\"Yes\"| ON2[Return TRUE
Enterprise gets early access]:::green\n R3 -->|\"No\"| R4\n\n R4 -->|\"hash < 25\"| ON3[Return TRUE
Part of rollout]:::green\n R4 -->|\"hash >= 25\"| R5\n\n R5 --> OFF2[Return FALSE]:::teal\n\n User --> R1\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nThis rule ordering implements a common pattern:\n\n1. Safety first: banned users are always excluded\n2. Internal testing: employees always see new features\n3. Premium access: enterprise customers get early access\n4. Gradual rollout: percentage of remaining users\n5. Default: everyone else gets the current behavior\n\n---\n\n# Performance Considerations\n\nFeature flags are checked on every request, sometimes multiple times per request. In a high-traffic system handling 100,000 requests per second, even small inefficiencies multiply into significant overhead.\n\n### The Performance Challenge\n\n\n```mermaid\nflowchart LR\n subgraph Scale[\"At Scale\"]\n S1[100K requests/second]:::primary\n S2[5 flag checks/request]:::primary\n S3[500K flag evaluations/second]:::red\n end\n\t\n subgraph Request[\"Single Request\"]\n\t\tdirection LR\n R1[Check auth flag]:::orange\n R2[Check feature flag 1]:::orange\n R3[Check feature flag 2]:::orange\n R4[Check experiment]:::orange\n R5[Check ops flag]:::orange\n end\t\n\n R1 --> R2 --> R3 --> R4 --> R5\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nIf each flag check requires a network call, you are adding 5-50ms of latency per check. At 500K evaluations per second, your flag service would need to handle massive load.\n\n### Local Evaluation with Caching\n\nThe solution is local evaluation: download all flag rules once, evaluate locally without network calls.\n\n\n```mermaid\nflowchart TD\n subgraph Startup[\"Application Startup\"]\n Init[\"SDK Initializes\"]:::primary\n Fetch[\"Fetch All Flag Rules
Single HTTP Request\"]:::primary\n Store[\"Store Rules in Memory\"]:::green\n end\n\n subgraph Runtime[\"Runtime (Every Request)\"]\n Check[\"flag.isEnabled(x, user)\"]:::orange\n Eval[\"Local Evaluation
No Network Call\"]:::green\n Result[\"Return Result
under 1ms\"]:::green\n end\n\n subgraph Updates[\"Background Updates\"]\n Stream[\"Streaming Connection
SSE/WebSocket\"]:::purple\n Update[\"Push Rule Changes\"]:::purple\n Refresh[\"Update Local Cache\"]:::green\n end\n\n Init --> Fetch --> Store\n Store --> Check --> Eval --> Result\n Stream --> Update --> Refresh --> Store\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n**Performance comparison:**\n\n\n| Approach | Latency per Check | Throughput | Reliability |\n|----------|-------------------|------------|-------------|\n| Remote evaluation | 10-50ms | Limited by network | Depends on flag service |\n| Local evaluation | < 1ms | Unlimited | Works even if flag service is down |\n\n\n### Caching Strategies\n\nDifferent caching levels provide different trade-offs:\n\n\n```mermaid\nflowchart TD\n App[\"Application\"]:::primary\n\n subgraph Caching[\"Cache Layers\"]\n L1[\"L1: In-Process Memory\\nTTL: 30s, Latency: ~0ms\"]:::green\n L2[\"L2: Distributed Cache (Redis)\\nTTL: 2min, Latency: 1-2ms\"]:::orange\n L3[\"L3: Flag Service API\\nLatency: 10-50ms\"]:::purple\n end\n\n App --> L1\n L1 -->|\"Miss\"| L2\n L2 -->|\"Miss\"| L3\n L3 --> L2\n L2 --> L1\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n\n| Cache Level | TTL | Trade-off |\n|-------------|-----|-----------|\n| **In-process** | 30s-2min | Fastest, but stale; memory per instance |\n| **Distributed** | 1-5min | Shared across instances, slight latency |\n| **CDN/Edge** | 5-15min | Global distribution, most stale |\n\n\n### Streaming Updates\n\nInstead of polling, receive updates in real-time via Server-Sent Events (SSE) or WebSockets:\n\n\n```mermaid\nsequenceDiagram\n participant App as Application\n participant SDK as Flag SDK\n participant Stream as Streaming Server\n\n App->>SDK: Initialize\n SDK->>Stream: Open SSE Connection\n Stream-->>SDK: Send all current flag rules\n\n Note over App,Stream: Connection stays open\n\n Note over Stream: Admin changes flag\n\n Stream-->>SDK: Push: flag 'new_checkout' updated\n SDK->>SDK: Update local cache\n Note over App: Next evaluation uses new rules
No polling delay\n\n Note over Stream: Another change\n\n Stream-->>SDK: Push: flag 'dark_mode' now 50%\n SDK->>SDK: Update local cache\n```\n\n\n**Comparison:**\n\n\n| Update Method | Propagation Delay | Complexity | Resource Usage |\n|---------------|-------------------|------------|----------------|\n| Polling (every 30s) | Up to 30 seconds | Simple | Regular network calls |\n| Streaming (SSE) | < 1 second | Medium | Persistent connections |\n| Streaming (WebSocket) | < 1 second | Higher | Bidirectional, more overhead |\n\n\nMost production systems use streaming with polling as a fallback in case the streaming connection drops.\n\n---\n\n# Lifecycle Management\n\nFeature flags accumulate. Without discipline, a codebase ends up with hundreds of flags, half of which no one remembers the purpose of, some protecting code that was removed months ago. This is feature flag debt, and it is real.\n\n### The Flag Lifecycle\n\n\n```mermaid\nstateDiagram-v2\n [*] --> Created: Create flag for new feature\n\n Created --> Development: Add flag checks to code\n Development --> InternalTesting: Enable for internal users\n InternalTesting --> GradualRollout: Enable for 10% of users\n\n GradualRollout --> GradualRollout: Increase to 25%, 50%, 75%\n GradualRollout --> FullyEnabled: Reach 100%\n GradualRollout --> Disabled: Issues found, rollback\n\n Disabled --> GradualRollout: Fixed, retry rollout\n\n FullyEnabled --> CleanupScheduled: Schedule flag removal\n CleanupScheduled --> Removed: Remove flag from code\n Removed --> [*]\n\n classDef created fill:#69db7c,stroke:#000,color:#000\n classDef dev fill:#ffa94d,stroke:#000,color:#000\n classDef testing fill:#ffd43b,stroke:#000,color:#000\n classDef rollout fill:#00ceff,stroke:#000,color:#000\n classDef enabled fill:#69db7c,stroke:#000,color:#000\n classDef cleanup fill:#9775fa,stroke:#000,color:#000\n classDef removed fill:#38d9a9,stroke:#000,color:#000\n\n class Created created\n class Development dev\n class InternalTesting testing\n class GradualRollout rollout\n class FullyEnabled enabled\n class CleanupScheduled cleanup\n class Removed removed\n```\n\n\n### Flag Metadata\n\nEvery flag should have metadata that answers: who owns this, why does it exist, and when should it be removed?\n\n\n```yaml\n# Example flag definition with metadata\nflag:\n key: new_checkout_flow\n name: New Checkout Flow\n description: Redesigned checkout with one-page experience\n\n # Ownership\n owner: checkout-team\n ownerEmail: checkout@company.com\n\n # Lifecycle\n type: release\n createdAt: 2024-01-15\n expectedRemovalDate: 2024-02-28\n jiraTicket: CHECKOUT-456\n\n # Current state\n status: rolling_out\n currentPercentage: 50\n environments:\n production: 50%\n staging: 100%\n development: 100%\n\n # Checklist\n rolloutChecklist:\n - [x] Code merged to main\n - [x] Internal testing complete\n - [x] 10% rollout (2024-01-20)\n - [x] 25% rollout (2024-01-22)\n - [x] 50% rollout (2024-01-25)\n - [ ] 100% rollout\n - [ ] Cleanup PR created\n - [ ] Flag removed from codebase\n```\n\n\n### Preventing Flag Debt\n\n\n```mermaid\nflowchart TD\n\n subgraph Bad[\"Flag Debt Symptoms\"]\n B1[Flags with no owner]:::red\n B2[Flags at 100% for months]:::red\n B3[Nested flag conditions]:::red\n B4[Flags protecting removed code]:::red\n B5[Hundreds of active flags]:::red\n end\n\t\n subgraph Good[\"Healthy Flag Hygiene\"]\n G1[Every flag has an owner]:::green\n G2[Every flag has an expiration date]:::green\n G3[Removal is tracked in backlog]:::green\n G4[Regular cleanup sprints]:::green\n G5[Flag count is a monitored metric]:::green\n end\t\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n#### **Best practices:**\n\n1. **Set expiration dates at creation time:** When you create a release flag, set an expected removal date (usually 2 weeks after planned 100% rollout)\n2. **Track removal as part of done:** A feature is not complete until the flag is removed. Add \"remove feature flag\" to your definition of done.\n3. **Monitor flag metrics:** Track total flag count, flags past expiration, and flags without owners as alerts\n4. **Schedule cleanup sprints:** Quarterly or monthly, dedicate time specifically to removing old flags\n5. **Limit flag creation:** Require approval for new flags, or limit creation to specific teams\n\n### Metrics to Monitor\n\nTrack these metrics to keep flag debt under control:\n\n\n| Metric | Healthy | Warning | Critical |\n|--------|---------|---------|----------|\n| Total active flags | < 50 | 50-100 | > 100 |\n| Flags past expiration | 0 | 1-5 | > 5 |\n| Flags without owner | 0 | 1-3 | > 3 |\n| Average flag age | < 30 days | 30-60 days | > 60 days |\n| Flags at 100% > 2 weeks | 0 | 1-3 | > 3 |\n\n\n---\n\n# Common Patterns\n\n### Trunk-Based Development\n\nFeature flags enable trunk-based development, where all developers merge to main frequently. Incomplete features are hidden behind flags.\n\n**Benefits:**\n\n- No long-lived feature branches\n- No merge conflicts from diverged branches\n- Continuous integration actually works\n- Code is always in a deployable state\n\n**The pattern:**\n\n\n```python\n# Incomplete feature merged to main, but safe\ndef process_order(order):\n if flags.is_enabled(\"new_order_processing\"):\n return new_order_processing(order) # Half done, but unreachable\n else:\n return old_order_processing(order) # Production path\n```\n\n\n### Dark Launches\n\nDeploy and run new code without users seeing results. Useful for validating new implementations:\n\n\n```mermaid\nflowchart TD\n Request([User Request]):::rose\n\n subgraph DarkLaunch[\"Dark Launch Pattern\"]\n Old[Old Service]:::green\n New[New Service
Flag Protected]:::primary\n\n Compare{Compare
Results}:::orange\n\n Old --> OldResult[Old Result]:::green\n New --> NewResult[New Result]:::primary\n\n OldResult --> Compare\n NewResult --> Compare\n\n Compare --> Log[Log Differences]:::purple\n end\n\n Return[Return Old Result
User sees nothing new]:::green\n\n Request --> Old\n Request --> New\n Compare --> Return\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n\n```python\ndef get_recommendations(user):\n # Always get old results\n old_result = old_recommendation_service.get(user)\n\n # Run new service in shadow mode\n if flags.is_enabled(\"dark_launch_new_recs\"):\n try:\n new_result = new_recommendation_service.get(user)\n if old_result != new_result:\n log.info(\"Recommendation mismatch\", old=old_result, new=new_result)\n except Exception as e:\n log.error(\"New service failed\", error=e)\n\n # Always return old result - users see no change\n return old_result\n```\n\n\n**Use cases:**\n\n- Validating new algorithm produces correct results\n- Testing new service handles production traffic\n- Building confidence before switching\n\n### Kill Switch Pattern\n\nUse ops flags as manual circuit breakers for non-critical dependencies:\n\n\n```mermaid\nflowchart TD\n Request([User Request]):::rose\n\n Check{recommendations
enabled?}:::orange\n\n subgraph Normal[\"Normal Operation\"]\n Call[Call Recommendation Service]:::green\n Timeout{Success?}:::orange\n Success[Return Personalized Results]:::green\n Error[Return Fallback]:::yellow\n end\n\n subgraph Incident[\"During Incident\"]\n Skip[Skip Service Call]:::yellow\n Fallback[Return Static Defaults]:::yellow\n end\n\n Request --> Check\n Check -->|\"ON\"| Call --> Timeout\n Timeout -->|\"Yes\"| Success\n Timeout -->|\"No\"| Error\n\n Check -->|\"OFF\"| Skip --> Fallback\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n```\n\n\nDuring an incident, on-call can disable the flag instantly to reduce load on the struggling service, without any deployment.\n\n### Operational Toggles\n\nControl system behavior dynamically without code changes:\n\n\n| Flag | Default | Purpose |\n|------|---------|---------|\n| enable_full_text_search | ON | Disable during high DB load |\n| enable_batch_processing | ON | Disable during peak traffic |\n| enable_detailed_logging | OFF | Enable for debugging |\n| cache_ttl_multiplier | 1.0 | Increase during incidents |\n| rate_limit_requests_per_second | 1000 | Decrease if backend overwhelmed |\n\n\nThese flags often have numeric values rather than boolean, giving operators fine-grained control over system behavior.\n\n---\n\n# Feature Flag Services\n\nWhen evaluating whether to build or buy a feature flag solution, consider your requirements and constraints.\n\n### Build vs Buy Matrix\n\n\n```mermaid\nflowchart TD\n Question{What do you need?}:::orange\n\n Question -->|\"Simple on/off\"| Build1[Build: Config file]:::green\n Question -->|\"User targeting\"| Buy1[Consider buying]:::primary\n Question -->|\"A/B testing\"| Buy2[Strongly consider buying]:::primary\n Question -->|\"Multiple platforms\"| Buy3[Buy: SDK maintenance is significant]:::purple\n\n Build1 --> Simple[Low complexity
No external deps]:::green\n Buy1 --> Medium[Medium complexity
Targeting rules engine]:::primary\n Buy2 --> High[High complexity
Statistical analysis]:::primary\n Buy3 --> VeryHigh[Very high complexity
SDK per platform]:::purple\n\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\n### When to Build\n\n\n| Scenario | Rationale |\n|----------|-----------|\n| Simple on/off flags only | Config file or environment variables suffice |\n| Strict compliance requirements | Data cannot leave your infrastructure |\n| Very high scale (millions of evaluations/second) | May need custom optimization |\n| Deep integration with existing systems | Custom solution might integrate better |\n| Limited budget, engineering time available | Open source self-hosted options exist |\n\n\n### When to Buy\n\n\n| Scenario | Rationale |\n|----------|-----------|\n| Need user targeting and segmentation | Building rules engine is significant effort |\n| Multiple platforms (web, iOS, Android, backend) | Maintaining SDKs is ongoing work |\n| A/B testing with statistical analysis | Statistical rigor is hard to get right |\n| Need audit logs and compliance features | Enterprise features take time to build |\n| Limited engineering time | Focus on your core product instead |\n\n\n### Options Landscape\n\n**Self-Hosted Open Source:**\n\n\n| Solution | Strengths |\n|----------|-----------|\n| **Unleash** | Full-featured, good for privacy-conscious teams |\n| **Flagsmith** | REST API first, cloud and self-hosted options |\n| **GrowthBook** | Strong A/B testing and experimentation focus |\n| **OpenFeature** | Vendor-neutral standard, multiple backends |\n\n\n**Managed Services:**\n\n\n| Service | Strengths |\n|---------|-----------|\n| **LaunchDarkly** | Industry leader, excellent SDKs, streaming updates |\n| **Split.io** | Strong experimentation and analytics |\n| **Statsig** | Deep analytics integration, ML-powered |\n| **ConfigCat** | Simple, affordable, good for small teams |\n| **Flagsmith Cloud** | Open-source with managed option |\n\n\n---\n\n# Risks and Anti-Patterns\n\nFeature flags are powerful, but misuse creates significant problems.\n\n### Flag Explosion\n\n\n```mermaid\nflowchart TD\n subgraph Problem[\"The Problem\"]\n P1[Start: 10 flags]:::green\n P2[6 months: 50 flags]:::yellow\n P3[1 year: 150 flags]:::orange\n P4[2 years: 400 flags]:::red\n end\n\n subgraph Symptoms[\"Symptoms\"]\n S1[Nobody knows what flags do]:::red\n S2[Impossible to test all combinations]:::red\n S3[Performance degradation]:::red\n S4[Fear of removing any flag]:::red\n end\n\n P1 --> P2 --> P3 --> P4\n P4 --> Symptoms\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Prevention**\n\n- Set expiration dates for all release flags\n- Track flag count as a metric with alerts\n- Schedule regular cleanup sprints\n- Require approval for new flags\n\n### Nested Flag Conditions\n\n\n```python\n# ANTI-PATTERN: Nested flags\nif flags.is_enabled(\"new_checkout\"):\n if flags.is_enabled(\"checkout_v2_payment\"):\n if flags.is_enabled(\"checkout_stripe\"):\n # What combination are we even testing?\n # 2^3 = 8 possible states\n process_with_stripe_v2()\n```\n\n\nWith 3 nested flags, you have 8 possible combinations. With 5 nested flags, you have 32 combinations. Testing becomes impossible.\n\n**Solution:** Use a single flag with variants, or restructure to avoid nesting:\n\n\n```python\n# BETTER: Single flag with variants\nvariant = flags.get_variant(\"checkout_version\")\nif variant == \"v2_stripe\":\n process_with_stripe_v2()\nelif variant == \"v2_paypal\":\n process_with_paypal_v2()\nelse:\n process_legacy()\n```\n\n\n### Testing Gaps\n\n\n```mermaid\nflowchart TD\n subgraph Mistake[\"Common Mistake\"]\n M1[Write tests with flag ON]:::green\n M2[Tests pass]:::green\n M3[Deploy to production]:::green\n M4[Flag is OFF for most users]:::yellow\n M5[Bug in OFF path]:::red\n M6[Users experience errors]:::red\n end\n\n M1 --> M2 --> M3 --> M4 --> M5 --> M6\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Solution:** Test both branches:\n\n\n```python\ndef test_checkout_new_path():\n with flags.override(\"new_checkout\", True):\n result = checkout(order)\n assert result.success\n\ndef test_checkout_old_path():\n with flags.override(\"new_checkout\", False):\n result = checkout(order)\n assert result.success\n```\n\n\n### Stale Flags\n\nA flag at 100% for months is technical debt:\n\n\n```python\n# This flag has been at 100% since March\nif flags.is_enabled(\"new_feature_march_2023\"):\n new_feature() # This is the only code path\nelse:\n old_feature() # Dead code that still exists\n```\n\n\nThe else branch is dead code, but:\n\n- It still needs to compile\n- It might still have tests\n- It confuses new team members\n- It increases cognitive load\n\n**Solution:** Remove flags within 2 weeks of reaching 100%:\n\n\n```python\n# After cleanup - simple and clear\nnew_feature()\n```\n\n\n---\n\n# Summary\n\nFeature flags provide runtime control over feature visibility, enabling safer and more flexible releases:\n\n**Core concept:** Conditional statements that check external configuration to determine what users see. This decouples deployment (putting code in production) from release (making features available to users).\n\n**Types of flags:**\n\n- Release flags: control feature rollouts (short-lived, remove after 100%)\n- Experiment flags: enable A/B testing (medium-lived, remove after experiment)\n- Ops flags: kill switches for operational control (permanent, never remove)\n- Permission flags: access control based on entitlements (tied to product model)\n\n**Targeting strategies:** Enable features for specific users, user segments based on attributes, or percentages of traffic using consistent hashing. Rules are evaluated in order with first match winning.\n\n**Performance:** Use local evaluation with cached rules to achieve sub-millisecond flag checks. Streaming updates push changes in real-time without polling delays.\n\n**Lifecycle management:** Flags accumulate as technical debt. Set expiration dates, track ownership, monitor metrics, and remove flags within two weeks of reaching 100%.\n\n**Common patterns:** Trunk-based development (merge incomplete features safely), dark launches (test new implementations without user visibility), and kill switches (instant operational control during incidents).\n\nFeature flags let you control who sees what at runtime. Combined with canary releases and rolling updates, you have a complete toolkit for managing deployment risk. But all these strategies share a common challenge: how do you know if the change is working? How do you measure whether the new version is better? This brings us to observability and metrics, the foundation for making data-driven deployment decisions.\n\n---\n\n### References\n\n- [Martin Fowler - Feature Toggles](undefined)\n- [LaunchDarkly - Feature Flag Best Practices](undefined)\n- [Unleash Documentation](undefined)\n- [Trunk Based Development - Feature Flags](undefined)\n- [OpenFeature Specification](undefined)","pageType":"system-design"}

Feature Flags

Ashish Pratap Singh

Get Premium