{"title":"Design Spotify","description":null,"content":"\n> **QUESTION**\n>\n> #### What is Spotify?\n>\n> [Spotify](https://open.spotify.com/) is a music streaming platform that allows users to search, discover, and listen to millions of songs and podcasts on demand. It provides personalized recommendations, curated playlists, offline downloads, and features like shuffle, repeat, and cross-device syncing.\n>\n> \n> \n> \n>\n> Users can follow artists, share playlists, and explore trending or new releases.\n>\n> With over **600 million** monthly active users (MAU) and **200 million** paid users, Spotify is the most popular music streaming platform in the world.\n>\n> **Other Popular Examples:** [Apple Music](https://www.apple.com/apple-music/), [Amazon Music](https://music.amazon.com), [YouTube Music](https://music.youtube.com)\n\n\nThis system design problem touches on several fundamental concepts: content delivery at global scale, real-time streaming protocols, search infrastructure, recommendation systems, and high availability architecture.\n\nIn this chapter, we will walk through the **high-level design of a music streaming platform like Spotify.**\n\nLet’s begin by clarifying the requirements.\n\n---\n\n# 1. Requirements Gathering\n\nBefore jumping into architecture diagrams, we need to understand what we are building. \n\nMusic streaming might seem straightforward, but the requirements can vary significantly. Are we designing for millions or hundreds of millions of users? Do we need to support offline playback? How sophisticated should the recommendation engine be? These questions shape our design decisions.\n\nHere is how a requirements discussion might unfold in an interview:\n\n\n> **DISCUSSION**\n>\n> **Candidate:** \"How many users should the system support, and what does daily usage look like?\"\n>\n> **Interviewer:** \"Let's design for 500 million total users, with about 200 million daily active users. On an average day, we see around 1 billion streams.\"\n>\n> **Candidate:** \"That's substantial scale. A billion streams per day means we are dealing with serious CDN and bandwidth requirements. Should I focus primarily on music, or do we need to handle podcasts and other audio formats as well?\"\n>\n> **Interviewer:** \"Focus on music streaming for now. Podcasts have different characteristics like longer duration and less repeat listening, so you can mention them but don't need to design for them in detail.\"\n>\n> **Candidate:** \"Understood. What about the user experience around playback? I am thinking about latency requirements and whether users can download content for offline listening.\"\n>\n> **Interviewer:** \"Latency is critical. Users expect to hear audio within 200ms of pressing play. As for offline playback, yes, that's an important feature for premium subscribers, especially for people with limited data plans or unreliable connectivity.\"\n>\n> **Candidate:** \"Personalization is a big part of what makes these services sticky. Should I include the recommendation system in the design, or treat it as out of scope?\"\n>\n> **Interviewer:** \"Cover the high-level approach to recommendations, though you don't need to dive deep into the ML algorithms.\"\n>\n> **Candidate:** \"One more question: should I design the content ingestion pipeline, meaning how artists upload music to the platform?\"\n>\n> **Interviewer:** \"That's a separate system entirely. Focus on the consumer-facing streaming experience.\"\n\n\nThis conversation reveals several important constraints that will influence our design. Let's formalize these into functional and non-functional requirements.\n\n\n### Functional Requirements\n\n1. **Search:** Users can search for songs, albums, artists, and playlists. Search must be fast (under 100ms) and handle typos, partial matches, and multiple languages.\n2. **Stream Music:** Users can play any song on-demand with minimal latency. The system should handle quality adaptation based on network conditions.\n3. **Playlist Management:** Users can create, edit, delete, and share playlists. Playlists can contain thousands of songs and be collaborative.\n4. **Personalized Recommendations:** The system provides personalized recommendations based on listening history, liked songs, and contextual signals like time of day.\n5. **Offline Playback:** Premium users can download songs and playlists for offline listening, with appropriate DRM protection.\n\n### Non-Functional Requirements\n\n1. **High Availability:** The system should target 99.99% uptime (roughly 52 minutes of downtime per year). Users expect music to be accessible whenever they want it, and outages are highly visible.\n2. **Low Latency:** Playback should start within 200ms of pressing play. Once playing, buffering should be rare even on variable network conditions.\n3. **Scalability:** The system must handle 1 billion+ streams per day across 200 million daily active users, with traffic patterns that spike during commute hours and weekends.\n4. **Global Reach:** Users expect low latency regardless of location. This requires edge infrastructure and CDN presence across major regions.\n5. **Durability:** User data like playlists, liked songs, and listening history must never be lost. Losing a user's carefully curated playlist would be unacceptable.\n\n\n---\n\n# 2. Back-of-the-Envelope Estimation\n\nBefore diving into the design, let's run some quick calculations to understand the scale we are dealing with. These numbers will guide our architectural decisions, particularly around CDN infrastructure, storage, and database selection.\n\n### 2.1 Traffic Estimates\n\nStarting with the numbers from our requirements discussion:\n\n#### Streaming Traffic\n\nWe expect 1 billion streams per day. Let's convert this to queries per second (QPS):\n\n\n```python\nAverage stream QPS = 1,000,000,000 streams / 86,400 seconds ≈ 11,500 QPS\n```\n\n\nTraffic is rarely uniform throughout the day. Music streaming sees significant spikes during morning and evening commutes, lunch breaks, and weekends. During peak hours, we might see 3x the average load:\n\n\n```python\nPeak stream QPS = 11,500 × 3 ≈ 35,000 QPS\n```\n\n\n#### Metadata Traffic\n\nEvery stream request triggers additional metadata lookups: song info, artist details, album art URLs, lyrics. Users also browse, search, and load playlists. This metadata traffic is typically 5-10x higher than the stream traffic:\n\n\n```python\nEstimated metadata QPS: 50,000 - 100,000 QPS\n```\n\n\n\n```mermaid\nflowchart LR\n subgraph \"Daily Traffic\"\n direction TB\n W[Streams
1B per day
11,500 QPS avg]:::orange\n R[Metadata
50K-100K QPS
searches, browses]:::primary\n end\n\n subgraph \"Peak Traffic (3x)\"\n direction TB\n WP[35,000 QPS
peak streams]:::orange\n RP[~300K QPS
peak metadata]:::primary\n end\n\n W --> WP\n R --> RP\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nThese numbers are significant. At peak, we are handling over 300,000 requests per second across all services. This is why we need aggressive caching at multiple layers.\n\n### 2.2 Storage Estimates\n\nMusic streaming has an interesting storage profile: a large catalog of audio files that rarely changes, combined with user data that grows continuously.\n\n\n```mermaid\nflowchart LR\n subgraph \"Storage Components\"\n direction TB\n A[Audio Files
1 PB total]:::teal\n B[Metadata
200 GB]:::primary\n C[User Data
7.5 TB]:::orange\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n#### Audio Files (The Big One)\n\nThe audio catalog is where most of the storage goes. We have 100 million songs in the catalog, each stored in multiple quality levels (64, 96, 160, 320 kbps), which works out to an average total size of around 10 MB per song across all qualities.\n\n\n```python\nTotal audio storage = 100,000,000 songs × 10 MB = 1 PB (Petabyte)\n```\n\n\nOne petabyte is substantial but manageable with object storage services like S3. The key insight is that this data is mostly static. Songs don't change once uploaded, which makes caching highly effective.\n\n#### Metadata\n\nSong metadata is relatively small but needs to be fast:\n\n\n| Data Type | Estimate | Notes |\n|-----------|----------|-------|\n| Song metadata | 100M × 2 KB = 200 GB | Title, artist, album, duration, genre |\n| Artist profiles | 10M × 5 KB = 50 GB | Bio, images, discography |\n| Album data | 20M × 3 KB = 60 GB | Track lists, artwork, release info |\n\n\nThis comfortably fits in memory for caching, which is important for the response times we need.\n\n#### User Data\n\nUser data grows continuously and needs different handling:\n\n\n| Data Type | Estimate | Notes |\n|-----------|----------|-------|\n| User profiles | 500M × 2 KB = 1 TB | Preferences, settings, subscription |\n| Playlists | 500M × 5 KB = 2.5 TB | Average user has 5-10 playlists |\n| Listening history | 500M × 8 KB = 4 TB | Last 6 months of plays |\n\n\n### 2.3 Bandwidth Estimates\n\nBandwidth is where music streaming gets expensive. Each stream transfers significant data:\n\n\n```python\nAverage song: 4 minutes at 160 kbps = ~5 MB\nDaily data transfer: 1B streams × 5 MB = 5 PB per day\n```\n\n\nFive petabytes per day of bandwidth is massive. At typical cloud egress rates ($0.05-0.09 per GB), this would cost millions per month without optimization. This is exactly why CDN infrastructure is not optional for music streaming. It is essential for both performance and cost control.\n\n\n```mermaid\nflowchart TB\n\n subgraph \"With CDN\"\n B1[Popular songs cached at edge]:::green\n B2[~200ms latency globally]:::green\n B3[80%+ requests served from cache]:::green\n end\n\t\n subgraph \"Without CDN\"\n A1[Every request goes to origin]:::red\n A2[High latency for distant users]:::red\n A3[Massive bandwidth costs]:::red\n end\t\n\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n### 2.4 Key Insights\n\nThese estimates reveal several important design implications:\n\n1. **CDN is critical:** With 5 PB of daily bandwidth, serving from origin is financially and technically impractical. Most streams must be served from edge caches.\n2. **Read-heavy workload:** For every song uploaded (which happens through a separate pipeline), there are millions of plays. We should invest heavily in caching and optimize for read performance.\n3. **Metadata fits in memory:** At 200 GB, song metadata can be cached aggressively. This enables the sub-100ms response times users expect.\n4. **Audio storage is static:** Unlike user-generated content platforms, the audio catalog changes slowly. This makes CDN caching straightforward.\n5. **User data is the scaling challenge:** While audio files are large but static, user listening history grows continuously and drives database scaling decisions.\n\n---\n\n# 3. Core APIs\n\nWith our requirements and scale understood, let's define the API contract. A music streaming service needs APIs for search, playback, playlist management, and recommendations. Let's walk through each one.\n\n\n```mermaid\nflowchart LR\n subgraph \"Core API Operations\"\n Search[GET /search
Find content]:::primary\n Stream[GET /songs/.../stream
Get audio URL]:::green\n Playlist[POST /playlists
Manage playlists]:::orange\n Recs[GET /recommendations
Get personalized picks]:::teal\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n### 3.1 Search\n\n#### Endpoint: `GET /search`\n\nThis is the entry point for discovery. Users type a query and expect relevant songs, artists, albums, and playlists back within milliseconds.\n\n#### Request Parameters\n\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `query` | string | Yes | The search term (e.g., \"bohemian rhapsody\", \"queen\") |\n| `type` | string | No | Filter by type: \"song\", \"artist\", \"album\", \"playlist\". Default: all types |\n| `limit` | integer | No | Number of results per type. Default: 20 |\n| `offset` | integer | No | Pagination offset for results |\n\n\n#### Example Response (200 OK)\n\n\n```json\n{\n \"songs\": [\n {\n \"id\": \"song_123\",\n \"title\": \"Bohemian Rhapsody\",\n \"artist\": \"Queen\",\n \"album\": \"A Night at the Opera\",\n \"duration_ms\": 354000,\n \"album_art_url\": \"https://cdn.example.com/art/album_456.jpg\"\n }\n ],\n \"artists\": [\n {\n \"id\": \"artist_789\",\n \"name\": \"Queen\",\n \"image_url\": \"https://cdn.example.com/artist/queen.jpg\",\n \"monthly_listeners\": 45000000\n }\n ],\n \"albums\": [...]\n}\n```\n\n\n### 3.2 Get Stream URL\n\n#### Endpoint: `GET /songs/{song_id}/stream`\n\nThis is the critical path for playback. Instead of streaming audio directly through our API servers, we return a signed URL that points to a CDN edge server. This approach offloads bandwidth from our application tier and ensures users get audio from a server close to them.\n\n#### Path Parameters\n\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `song_id` | string | The unique identifier for the song |\n\n\n#### Query Parameters\n\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `quality` | string | No | Preferred quality: \"low\" (96kbps), \"medium\" (160kbps), \"high\" (320kbps)|\n\n\n#### Example Response (200 OK)\n\n\n```json\n{\n \"stream_url\": \"https://cdn.example.com/audio/song_123/320.mp3?token=eyJhbGc...&expires=1705320000\",\n \"expires_at\": \"2024-01-15T12:00:00Z\",\n \"format\": \"mp3\",\n \"bitrate\": 320,\n \"duration_ms\": 354000\n}\n```\n\n\nThe signed URL includes an authentication token and expiration time. This prevents URL sharing (the token is tied to the user) and limits the window for unauthorized access.\n\n### 3.3 Create Playlist\n\n#### Endpoint: `POST /playlists`\n\nCreates a new playlist for the authenticated user.\n\n#### Request Body\n\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `name` | string | Yes | Playlist name (max 100 characters) |\n| `description` | string | No | Playlist description |\n| `is_public` | boolean | No | Whether the playlist is publicly visible. Default: true |\n\n\n#### Example Request\n\n\n```json\n{\n \"name\": \"My Workout Mix\",\n \"description\": \"High energy songs for the gym\",\n \"is_public\": true\n}\n```\n\n\n#### Example Response (201 Created)\n\n\n```json\n{\n \"playlist_id\": \"playlist_456\",\n \"name\": \"My Workout Mix\",\n \"description\": \"High energy songs for the gym\",\n \"owner_id\": \"user_789\",\n \"is_public\": true,\n \"created_at\": \"2024-01-15T10:00:00Z\",\n \"song_count\": 0\n}\n```\n\n\n### 3.4 Add Songs to Playlist\n\n#### Endpoint: `POST /playlists/{playlist_id}/songs`\n\nAdds one or more songs to an existing playlist. Songs are added at the end of the playlist by default.\n\n#### Request Body\n\n\n```json\n{\n \"song_ids\": [\"song_123\", \"song_456\", \"song_789\"],\n \"position\": 0\n}\n```\n\n\nThe optional `position` parameter allows inserting songs at a specific index. If omitted, songs are appended.\n\n#### Example Response (200 OK)\n\n\n```json\n{\n \"playlist_id\": \"playlist_456\",\n \"added_count\": 3,\n \"song_count\": 15\n}\n```\n\n\n### 3.5 Get Recommendations\n\n#### Endpoint: `GET /recommendations`\n\nReturns personalized song recommendations based on the user's listening history and optional seed inputs.\n\n#### Query Parameters\n\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `seed_songs` | string | No | Comma-separated song IDs to base recommendations on |\n| `seed_artists` | string | No | Comma-separated artist IDs |\n| `limit` | integer | No | Number of recommendations (1-100). Default: 20 |\n\n\n#### Example Response (200 OK)\n\n\n```json\n{\n \"songs\": [\n {\n \"id\": \"song_999\",\n \"title\": \"Don't Stop Me Now\",\n \"artist\": \"Queen\",\n \"album\": \"Jazz\",\n \"duration_ms\": 210000,\n \"reason\": \"Because you listened to Bohemian Rhapsody\"\n },\n {\n \"id\": \"song_888\",\n \"title\": \"Under Pressure\",\n \"artist\": \"Queen & David Bowie\",\n \"album\": \"Hot Space\",\n \"duration_ms\": 248000,\n \"reason\": \"Popular with Queen fans\"\n }\n ]\n}\n```\n\n\nThe `reason` field provides transparency about why each song was recommended, which helps users understand and trust the recommendations.\n\n---\n\n# 4. High-Level Design\n\nNow we get to the interesting part: designing the system architecture. Rather than presenting a complex diagram upfront, we will build the design incrementally, starting with the core user journey (streaming music) and adding components as we address each requirement. This mirrors how you would approach the problem in an interview.\n\nOur system needs to handle four core operations:\n\n1. **Stream Music:** Play any song with sub-200ms latency\n2. **Search:** Find songs, artists, and albums instantly\n3. **Manage Playlists:** Create, edit, and share playlists\n4. **Get Recommendations:** Surface personalized music discovery\n\nThe system is heavily read-intensive. For every song added to the catalog (which happens through a separate ingestion pipeline), there are millions of streams. This read-to-write asymmetry means we should optimize aggressively for read performance, and caching will be essential at every layer.\n\nLet's visualize the two primary paths through our system:\n\n\n```mermaid\nflowchart LR\n subgraph \"Stream Path (most traffic)\"\n direction LR\n S1[Client]:::rose --> S2[API Gateway]:::secondary\n S2 --> S3[Playback Service]:::primary\n S3 --> S4[Generate Signed URL]:::orange\n S1 -.->|direct audio fetch| S5[CDN Edge]:::green\n S5 -.->|cache miss| S6[(Object Storage)]:::teal\n end\n\n subgraph \"Data Path (search, browse, playlists)\"\n direction LR\n D1[Client]:::rose --> D2[API Gateway]:::secondary\n D2 --> D3[Application Services]:::primary\n D3 --> D4[Cache Layer]:::orange\n D4 -.->|miss| D5[(Databases)]:::teal\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nNotice how the stream path separates the control plane (getting a signed URL) from the data plane (actual audio delivery via CDN). This separation is key to handling billions of streams without our API servers becoming a bottleneck.\n\nLet's build this architecture step by step.\n\n---\n\n## 4.1 Requirement 1: Music Streaming\n\nThis is the core functionality. When a user taps play, audio must start within 200ms. Once playing, buffering should be rare even on variable network conditions. Let's design for this critical path first.\n\nWhen a user clicks play on a song, several things need to happen:\n\n1. Validate the user has permission to play this song (subscription, region)\n2. Generate a secure, time-limited URL for the audio file\n3. Return the URL so the client can fetch audio directly from a CDN edge server\n4. Track the play for analytics and royalty calculations\n\nLet's introduce the components we need to make this work.\n\n### Components for Streaming\n\nFour components work together to deliver audio from origin storage to a user's device in under 200ms: the API Gateway, the Playback Service, a CDN edge layer, and object storage holding the audio files.\n\n\n```mermaid\nflowchart TB\n U[User]:::rose\n\n subgraph Gateway[\"Gateway Layer\"]\n AG[API Gateway]:::secondary\n end\n\n subgraph Application[\"Application Layer\"]\n PS[Playback Service]:::primary\n end\n\n subgraph Edge[\"Edge Layer\"]\n CDN[CDN Edge Servers
Distributed Globally]:::green\n end\n\n subgraph Storage[\"Storage Layer\"]\n S3[(Object Storage
Audio Files)]:::teal\n end\n\n U -->|\"1. GET /songs/id/stream\"| AG\n AG -->|\"2. Validate + generate URL\"| PS\n PS -->|\"3. Return signed URL\"| AG\n AG -->|\"4. Return to client\"| U\n U -->|\"5. Fetch audio directly\"| CDN\n CDN -->|\"6. Cache miss? Fetch from origin\"| S3\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### API Gateway\n\nEvery request enters through the API Gateway. Think of it as the front door to our system, handling concerns that are common across all requests.\n\nThe gateway terminates SSL connections, validates request format, enforces rate limits, authenticates users via tokens, and routes requests to the appropriate backend service. By handling these cross-cutting concerns at the edge, we keep our application services focused on business logic.\n\n#### Playback Service\n\nThis service handles the critical decision: should this user be allowed to play this song? It checks:\n\n- Is the user authenticated with a valid session?\n- Does their subscription tier allow the requested quality (320kbps is premium-only)?\n- Is the song available in the user's geographic region (licensing varies by country)?\n- Has the user exceeded any rate limits?\n\nIf all checks pass, the service generates a signed URL that grants temporary access to the audio file on the CDN. The URL includes an authentication token and expiration time, typically 15-30 minutes. This prevents URL sharing and limits the window for unauthorized access.\n\n#### Content Delivery Network (CDN)\n\nThis is where the magic happens for low-latency playback. A CDN is a network of edge servers distributed globally. When a user in Tokyo requests audio, instead of fetching from our origin in Virginia, they get it from a nearby edge server.\n\nFor popular songs (think Taylor Swift's latest release), the audio is already cached at edge locations worldwide. The CDN can serve millions of concurrent streams without touching our origin servers. This is essential given our bandwidth estimates of 5 PB per day.\n\n#### Object Storage\n\nThe actual audio files live in object storage (S3, Google Cloud Storage, or Azure Blob). Files are organized by song ID and quality level:\n\n\n```python\ns3://audio-bucket/\n songs/\n song_abc123/\n 320kbps.mp3 (high quality, ~10 MB)\n 160kbps.mp3 (standard, ~5 MB)\n 96kbps.mp3 (low bandwidth, ~3 MB)\n manifest.json (metadata about available formats)\n```\n\n\nEach song exists in multiple quality levels to support adaptive bitrate streaming and different subscription tiers.\n\n### The Streaming Flow in Action\n\nThe sequence diagram traces a full play request from the moment the user taps play through authorization, signed URL generation, and audio delivery from the CDN.\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant API as API Gateway\n participant PS as Playback Service\n participant Cache as Redis Cache\n participant CDN as CDN Edge\n participant S3 as Object Storage\n\n Client->>API: GET /songs/song_123/stream\n Note over API: Validate auth token\n API->>PS: Forward with user context\n PS->>Cache: Get song metadata\n Cache-->>PS: Return metadata (or cache miss)\n Note over PS: Check subscription,
region, entitlements\n PS->>PS: Generate signed URL
(15 min expiry)\n PS-->>API: Return stream_url\n API-->>Client: 200 OK with signed URL\n\n Note over Client: Client uses signed URL directly\n Client->>CDN: GET /audio/song_123/320.mp3?token=...\n alt CDN Cache Hit\n CDN-->>Client: Stream audio (fastest path)\n else CDN Cache Miss\n CDN->>S3: Fetch from origin\n S3-->>CDN: Return audio file\n Note over CDN: Cache for future requests\n CDN-->>Client: Stream audio\n end\n```\n\n\nLet's trace through this step by step:\n\n1. **Client requests playback:** The user taps play. The client sends a request to our API Gateway with the song ID and the user's auth token.\n2. **Gateway validates and routes:** The API Gateway verifies the token is valid, checks rate limits, and forwards the request to the Playback Service with the user's context.\n3. **Playback Service checks entitlements:** The service looks up the song metadata (often from cache) and verifies the user can play this song. Premium-only songs require a premium subscription. Some songs are not available in certain countries due to licensing.\n4. **Generate signed URL:** If all checks pass, the service generates a signed URL. This URL includes a cryptographic signature that proves the user was authorized at this moment. The signature expires after 15-30 minutes.\n5. **Client fetches audio from CDN:** The client now has a URL pointing to a CDN edge server. It fetches the audio directly, bypassing our API entirely. This is crucial for scale.\n6. **CDN serves or fetches:** If the song is popular, it's already cached at the edge. If not, the CDN fetches from origin (S3), caches it, and serves. Either way, the audio streams to the client.\n\n**Why this design works:** By separating the control plane (authorization and URL generation) from the data plane (audio delivery), we can handle billions of streams. Our API servers handle lightweight authorization requests, while the heavy lifting of audio delivery falls to CDN infrastructure designed for exactly this purpose.\n\n---\n\n## 4.2 Requirement 2: Music Search\n\nUsers need to find any song instantly. They type \"bohemian\" and expect to see Queen's Bohemian Rhapsody appear within milliseconds. Search must be fast (under 100ms), handle typos (\"bohemain\"), partial matches, and return results ranked by relevance.\n\n### Components for Search\n\nSearch traffic flows through the API Gateway to a dedicated Search Service, which checks a Redis cache before falling back to an Elasticsearch cluster that holds a full-text index of the catalog.\n\n\n```mermaid\nflowchart LR\n U[User]:::rose\n\n subgraph Gateway[\"Gateway Layer\"]\n AG[API Gateway]:::secondary\n end\n\n subgraph Application[\"Application Layer\"]\n SS[Search Service]:::primary\n end\n\n subgraph Search[\"Search Infrastructure\"]\n ES[(Elasticsearch Cluster)]:::teal\n end\n\n subgraph Cache[\"Cache Layer\"]\n Redis[(Redis)]:::orange\n end\n\n U -->|\"1. GET /search?q=bohemian\"| AG\n AG -->|\"2. Route to search\"| SS\n SS -->|\"3. Check cache\"| Redis\n SS -->|\"4. Query index\"| ES\n SS -->|\"5. Return results\"| AG\n AG --> U\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n#### Search Service\n\nThis service handles all search queries. It does not query the primary database directly. Instead, it queries a dedicated search index optimized for text matching.\n\nThe Search Service parses queries, handles autocomplete requests, applies filters, and ranks results based on relevance and popularity. It also caches frequent queries since many users search for the same popular songs and artists.\n\n#### Elasticsearch Cluster\n\nWe use Elasticsearch (or a similar search engine like OpenSearch) for full-text search. Elasticsearch is purpose-built for this use case:\n\n- **Inverted indexes** enable fast text matching across 100 million songs\n- **Fuzzy matching** handles typos (\"bohemain\" matches \"bohemian\")\n- **Relevance scoring** ranks results by how well they match the query\n- **Horizontal scalability** allows us to add nodes as the catalog grows\n\nThe search index contains denormalized data: song titles, artist names, album names, genres, and popularity scores. When a new song is added to the catalog, an indexing pipeline updates Elasticsearch.\n\n### The Search Flow in Action\n\nThe sequence diagram shows how a cache hit short-circuits the round trip to Elasticsearch, and how a cache miss triggers an Elasticsearch query followed by caching the result for subsequent identical searches.\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant API as API Gateway\n participant SS as Search Service\n participant Cache as Redis\n participant ES as Elasticsearch\n\n Client->>API: GET /search?q=bohemian\n API->>SS: Forward search request\n SS->>Cache: Check if query cached\n alt Cache Hit\n Cache-->>SS: Return cached results\n else Cache Miss\n SS->>ES: Query: \"bohemian\"\n Note over ES: Fuzzy match,
rank by relevance
and popularity\n ES-->>SS: Return ranked results\n SS->>Cache: Cache results (5 min TTL)\n end\n SS-->>API: Return search results\n API-->>Client: 200 OK with songs, artists, albums\n```\n\n\n**Why cache search results?** Popular searches are repeated constantly. \"Taylor Swift\", \"Drake\", and \"Bohemian Rhapsody\" are searched thousands of times per minute. Caching these results for even a few minutes dramatically reduces load on Elasticsearch.\n\n---\n\n## 4.3 Requirement 3: Playlist Management\n\nUsers create playlists to organize their music. A playlist can contain anywhere from a few songs to thousands. Users expect to add songs instantly and have changes sync across all their devices.\n\n### Components for Playlists\n\nPlaylist writes touch the Playlist Service, a Redis cache for ownership verification and cache invalidation, and a persistent Playlist Database that keeps the authoritative record of each playlist's contents.\n\n\n```mermaid\nflowchart LR\n U[User]:::rose\n\n subgraph Gateway[\"Gateway Layer\"]\n AG[API Gateway]:::secondary\n end\n\n subgraph Application[\"Application Layer\"]\n PLS[Playlist Service]:::primary\n end\n\n subgraph Data[\"Data Layer\"]\n DB[(Playlist Database)]:::teal\n Cache[(Redis Cache)]:::orange\n end\n\n U -->|\"1. POST /playlists/id/songs\"| AG\n AG -->|\"2. Route request\"| PLS\n PLS -->|\"3. Verify ownership\"| Cache\n PLS -->|\"4. Add song\"| DB\n PLS -->|\"5. Invalidate cache\"| Cache\n PLS -->|\"6. Confirm\"| AG\n AG --> U\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n#### Playlist Service\n\nThis service manages all playlist CRUD operations. It creates, updates, and deletes playlists, adds and removes songs while maintaining their order, handles collaborative playlists where multiple users can edit, and manages sharing and privacy settings.\n\n#### Playlist Database\n\nWe need a database that handles high read volume, since users constantly load their playlists, along with a reasonable write volume from adding and removing songs. It also needs efficient queries by `user_id` (show me all my playlists) and efficient ordering, since songs have a position in the playlist.\n\nWe will discuss the specific database choice in the Database Design section.\n\n### The Playlist Flow in Action\n\nThe sequence diagram covers adding a song to a playlist: ownership verification from cache, the database write, and the cache invalidation that keeps all of a user's devices in sync.\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant API as API Gateway\n participant PLS as Playlist Service\n participant Cache as Redis\n participant DB as Playlist DB\n\n Client->>API: POST /playlists/pl_123/songs
{song_ids: [\"song_456\"]}\n API->>PLS: Forward with user context\n PLS->>Cache: Get playlist metadata\n Note over PLS: Verify user owns
this playlist\n PLS->>DB: Insert song at position\n DB-->>PLS: Confirm insert\n PLS->>Cache: Invalidate playlist cache\n PLS-->>API: Return updated playlist\n API-->>Client: 200 OK\n```\n\n\nThe cache invalidation is important: when a user adds a song on their phone, the change must appear immediately when they open the app on their laptop. By invalidating the cache, we ensure the next request fetches fresh data.\n\n---\n\n## 4.4 Requirement 4: Personalized Recommendations\n\nThis is what makes modern streaming services sticky. Users expect the app to know their taste and surface music they will love but haven't discovered yet. Features like \"Discover Weekly\" and \"Daily Mix\" drive significant engagement.\n\nRecommendations combine two fundamentally different patterns: batch processing (generating recommendations offline) and real-time serving (delivering them instantly when requested).\n\n### Components for Recommendations\n\nThe recommendation system splits into two paths: an offline batch pipeline that trains models and pre-computes results, and a real-time serving path that reads from Redis cache or the Feature Store on cache miss.\n\n\n```mermaid\nflowchart TB\n subgraph \"Batch Path (runs daily/weekly)\"\n direction LR\n Events[Listening Events]:::rose --> Pipeline[ML Pipeline]:::green\n Pipeline --> Features[(Feature Store)]:::teal\n Pipeline --> Recs[(Pre-computed
Recommendations)]:::teal\n end\n\n subgraph \"Serving Path (real-time)\"\n direction LR\n Client[Client Request]:::rose --> RS[Recommendation Service]:::primary\n RS --> Cache[(Redis Cache)]:::orange\n Cache -.->|miss| Recs\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n#### ML Pipeline (Offline)\n\nThis batch processing system runs periodically (daily or weekly). It aggregates listening history across all users, trains collaborative filtering models (users who liked X also liked Y), generates user taste profiles from audio features, and pre-computes personalized recommendations for each user.\n\nThe pipeline produces recommendations that are stored and ready to serve. This is how features like \"Discover Weekly\" work: they are generated once per week and cached.\n\n#### Feature Store\n\nThis stores computed features about users and songs: user taste vectors that capture preferred genres, moods, and tempos, song embeddings that numerically represent audio characteristics, and contextual features such as what users listen to in the morning versus the evening.\n\n#### Recommendation Service\n\nServes recommendations in real-time. For most requests, it fetches pre-computed recommendations from cache. For seed-based recommendations (\"songs similar to X\"), it may compute results on the fly using the feature store.\n\n### The Recommendation Flow in Action\n\nThe sequence diagram shows the common case, where pre-computed recommendations are returned from Redis in a single step, and the slower fallback path, where the Recommendation Service queries the Feature Store and ranks candidates before responding.\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant API as API Gateway\n participant RS as Recommendation Service\n participant Cache as Redis\n participant FS as Feature Store\n\n Client->>API: GET /recommendations (home page)\n API->>RS: Get personalized recommendations\n RS->>Cache: Check for pre-computed recs\n alt Cache Hit (common case)\n Cache-->>RS: Return recommendations\n else Cache Miss\n RS->>FS: Get user taste profile\n FS-->>RS: Return profile + candidate songs\n Note over RS: Rank candidates
by relevance\n RS->>Cache: Store results\n end\n RS-->>API: Return personalized songs\n API-->>Client: 200 OK with recommendations\n```\n\n\nThe key insight is that most recommendations are pre-computed. \"Discover Weekly\" is generated once per week for each user. When a user opens the app, we fetch their pre-computed playlist from cache. This approach allows us to use sophisticated ML models that would be too slow to run in real-time.\n\n---\n\n## 4.5 Putting It All Together\n\nNow that we have designed each requirement, let's step back and see the complete architecture:\n\n\n```mermaid\nflowchart TB\n subgraph Clients[\"Client Layer\"]\n direction LR\n Mobile[Mobile App]:::rose\n Web[Web Browser]:::rose\n Desktop[Desktop App]:::rose\n end\n\n subgraph Edge[\"Edge Layer\"]\n CDN[CDN
Audio Delivery]:::green\n LB[Load Balancer]:::secondary\n end\n\n subgraph Gateway[\"Gateway Layer\"]\n AG[API Gateway]:::secondary\n end\n\n subgraph Services[\"Application Layer\"]\n direction LR\n PS[Playback
Service]:::primary\n SS[Search
Service]:::primary\n PLS[Playlist
Service]:::primary\n RS[Recommendation
Service]:::primary\n US[User
Service]:::primary\n end\n\n subgraph Caching[\"Cache Layer\"]\n Redis[(Redis Cluster)]:::orange\n end\n\n subgraph Data[\"Data Layer\"]\n ES[(Elasticsearch)]:::teal\n MetaDB[(Metadata DB)]:::teal\n PlaylistDB[(Playlist DB)]:::teal\n UserDB[(User DB)]:::teal\n end\n\n subgraph Storage[\"Storage Layer\"]\n S3[(Object Storage
Audio Files)]:::teal\n end\n\n subgraph ML[\"ML Infrastructure\"]\n Pipeline[ML Pipeline]:::lightblue\n FeatureStore[(Feature Store)]:::lightblue\n end\n\n Mobile --> CDN\n Mobile --> LB\n Web --> LB\n Desktop --> LB\n CDN --> S3\n LB --> AG\n\n AG --> PS\n AG --> SS\n AG --> PLS\n AG --> RS\n AG --> US\n\n PS --> Redis\n PS --> MetaDB\n SS --> Redis\n SS --> ES\n PLS --> Redis\n PLS --> PlaylistDB\n RS --> Redis\n RS --> FeatureStore\n US --> UserDB\n\n Pipeline --> FeatureStore\n Pipeline --> Redis\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef lightblue fill:#3bc9db,stroke:#000,color:#000\n```\n\n\nThe architecture follows a layered approach, with each layer having a specific responsibility:\n\n**Client Layer:** Users access the service through mobile apps, web browsers, or desktop applications. From our perspective, they all look the same: HTTP requests with authentication tokens.\n\n**Edge Layer:** The CDN handles audio delivery at the edge, close to users geographically. The load balancer distributes API traffic across gateway instances.\n\n**Gateway Layer:** The API Gateway handles authentication, rate limiting, and request routing. It is the single entry point for all API traffic.\n\n**Application Layer:** Stateless services implement the business logic. Each service is independently deployable and horizontally scalable.\n\n**Cache Layer:** Redis provides low-latency access to frequently requested data: song metadata, user sessions, search results, and pre-computed recommendations.\n\n**Data Layer:** Different databases for different access patterns: Elasticsearch for search, relational databases for metadata and playlists, and potentially Cassandra for high-write-volume data like listening history.\n\n**ML Infrastructure:** Offline pipelines train models and generate recommendations. The feature store provides fast access to user and song features.\n\n### Component Responsibilities\n\nEach service in the architecture has a distinct role and a corresponding scaling strategy, ranging from managed services that scale automatically to stateless application tiers that add horizontal pods under load.\n\n\n| Component | Primary Responsibility | Scaling Strategy |\n|-----------|----------------------|------------------|\n| CDN | Audio delivery, edge caching | Managed service (auto-scales) |\n| Load Balancer | Traffic distribution | Managed service |\n| API Gateway | Auth, rate limiting, routing | Horizontal (stateless) |\n| Playback Service | Stream authorization, URL signing | Horizontal (stateless) |\n| Search Service | Query parsing, result ranking | Horizontal (stateless) |\n| Playlist Service | Playlist CRUD operations | Horizontal (stateless) |\n| Recommendation Service | Serve personalized content | Horizontal (stateless) |\n| Elasticsearch | Full-text search | Add nodes to cluster |\n| Redis Cluster | Hot data caching | Add nodes, shard by key |\n| Databases | Persistent storage | Read replicas, sharding |\n| Object Storage | Audio files | Managed (infinite scale) |\n\n\nThis architecture handles our requirements well: the CDN absorbs audio streaming traffic, the cache layer handles most read traffic, and the stateless services scale horizontally for everything else.\n\n---\n\n# 5. Database Design\n\nWith the high-level architecture in place, let's zoom into the data layer. Choosing the right database and designing an efficient schema are critical decisions that affect performance, scalability, and operational complexity.\n\n## 5.1 Choosing the Right Databases\n\nMusic streaming has diverse data with very different access patterns. Song metadata needs complex queries and relationships. Playlists need fast reads by user. Listening history is append-heavy with billions of writes. No single database excels at all of these.\n\nThis is where polyglot persistence shines: using different databases for different data types, each optimized for its specific access pattern.\n\n\n```mermaid\nflowchart LR\n subgraph \"Data Types and Their Databases\"\n Songs[Song Metadata
PostgreSQL]:::primary\n Search[Search Index
Elasticsearch]:::teal\n Playlists[Playlists
Cassandra]:::orange\n Sessions[User Sessions
Redis]:::green\n History[Listening History
Cassandra]:::orange\n Recs[Recommendations
Redis]:::green\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nLet's think through why each data type maps to a specific database:\n\n#### Song Metadata: PostgreSQL\n\nSong, artist, and album data has natural relationships: songs belong to albums, albums belong to artists, artists can collaborate on songs. A relational database handles these relationships well with foreign keys and joins.\n\nWe need complex queries: find all songs by this artist, find albums released in this year, get the top 50 songs in this genre. PostgreSQL's rich query language handles this naturally.\n\nThe catalog (100 million songs) changes slowly and fits comfortably in a single database with read replicas. No need for the complexity of sharding.\n\n#### Search Index: Elasticsearch\n\nWe already discussed this: full-text search, fuzzy matching, relevance scoring. Elasticsearch is purpose-built for these requirements. It sits alongside PostgreSQL, receiving updates when the catalog changes.\n\n#### Playlists: Cassandra\n\nPlaylist access patterns are user-centric: show me all my playlists, load this specific playlist, add a song to this playlist. These queries always include a user_id, making it a perfect partition key.\n\nCassandra excels here because it partitions data by user, so all of a user's playlists live on the same node, and reads and writes within a partition are fast. It also scales horizontally as the user count grows and handles the high write throughput of active playlist editing.\n\n#### User Sessions and Recommendations: Redis\n\nSession data (is this user logged in?) and pre-computed recommendations need sub-millisecond reads. Redis keeps this data in memory and handles millions of reads per second.\n\nRedis also supports TTLs natively, which is perfect for session expiration and recommendation refresh cycles.\n\n#### Listening History: Cassandra\n\nThis is the highest-write-volume data in the system. With 1 billion streams per day, we are writing 11,500+ events per second. Cassandra handles this write throughput while still supporting efficient reads by user (show me my recent listening history).\n\nThe partition key is user_id, and the clustering key is listened_at timestamp (descending). This means we can efficiently query \"last 100 songs this user played\" without scanning the entire table.\n\n### Database Choice Summary\n\n\n| Data Type | Database | Key Reasoning |\n|-----------|----------|---------------|\n| Song Metadata | PostgreSQL | Complex relationships, rich queries, stable catalog |\n| Search Index | Elasticsearch | Full-text search, fuzzy matching, relevance scoring |\n| Playlists | Cassandra | User-partitioned, high read/write throughput |\n| User Sessions | Redis | Sub-millisecond reads, TTL support |\n| Listening History | Cassandra | High write volume, time-series access pattern |\n| Recommendations | Redis | Pre-computed, needs instant reads |\n\n\n## 5.2 Database Schema\n\nNow let's design the schema for each data store.\n\n### Songs Table (PostgreSQL)\n\nThis is the heart of our catalog. Each row represents one song.\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `song_id` | UUID (PK) | Unique identifier for the song |\n| `title` | VARCHAR(255) | Song title |\n| `artist_id` | UUID (FK) | Reference to artists table |\n| `album_id` | UUID (FK) | Reference to albums table |\n| `duration_ms` | INTEGER | Song duration in milliseconds |\n| `release_date` | DATE | When the song was released |\n| `genres` | VARCHAR[] | Array of genre tags |\n| `audio_path` | VARCHAR(500) | Path to audio in object storage (e.g., \"songs/abc123/\") |\n| `play_count` | BIGINT | Total play count (updated periodically) |\n| `created_at` | TIMESTAMP | When the record was created |\n\n\n**Indexes:**\n\n\n```sql\n-- Primary lookup\nPRIMARY KEY (song_id)\n\n-- Find songs by artist\nCREATE INDEX idx_songs_artist ON songs(artist_id);\n\n-- Find songs by album\nCREATE INDEX idx_songs_album ON songs(album_id);\n\n-- Search by title (for exact match, fuzzy search uses Elasticsearch)\nCREATE INDEX idx_songs_title ON songs(title);\n```\n\n\n### Artists Table (PostgreSQL)\n\nArtist records store profile data that changes infrequently: name, biography, image URL, and a periodically refreshed monthly listener count used for search ranking.\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `artist_id` | UUID (PK) | Unique identifier for the artist |\n| `name` | VARCHAR(255) | Artist name |\n| `bio` | TEXT | Artist biography |\n| `image_url` | VARCHAR(500) | Profile image URL on CDN |\n| `monthly_listeners` | INTEGER | Current monthly listener count (updated daily) |\n| `verified` | BOOLEAN | Whether artist is verified |\n| `created_at` | TIMESTAMP | When the record was created |\n\n\n### Playlists Table (Cassandra)\n\nCassandra tables are designed around query patterns. Our primary queries are:\n\n- Get all playlists for a user\n- Get a specific playlist by ID\n- Get songs in a playlist\n\n\n```sql\n-- Playlists by user\nCREATE TABLE playlists_by_user (\n user_id UUID,\n playlist_id UUID,\n name TEXT,\n description TEXT,\n is_public BOOLEAN,\n song_count INT,\n created_at TIMESTAMP,\n updated_at TIMESTAMP,\n PRIMARY KEY (user_id, created_at, playlist_id)\n) WITH CLUSTERING ORDER BY (created_at DESC, playlist_id ASC);\n```\n\n\nThe partition key is `user_id`, so all of a user's playlists are on the same node. The clustering key orders playlists by creation date (newest first).\n\n### Playlist Songs Table (Cassandra)\n\nThis table stores the ordered contents of each playlist. Partitioning by `playlist_id` and clustering by `position` lets Cassandra return all songs in correct order with a single partition read.\n\n\n```sql\n-- Songs in a playlist\nCREATE TABLE playlist_songs (\n playlist_id UUID,\n position INT,\n song_id UUID,\n added_at TIMESTAMP,\n added_by UUID,\n PRIMARY KEY (playlist_id, position)\n) WITH CLUSTERING ORDER BY (position ASC);\n```\n\n\nThe partition key is `playlist_id`, and songs are ordered by position. This makes loading a playlist efficient: one query returns all songs in order.\n\n### User Listening History Table (Cassandra)\n\nListening history is the highest-write-volume table in the system. Partitioning by `user_id` and clustering by `listened_at` in descending order makes it fast to retrieve a user's most recent plays, which feed both the recommendation pipeline and the listening history UI.\n\n\n```sql\n-- Listening history by user\nCREATE TABLE listening_history (\n user_id UUID,\n listened_at TIMESTAMP,\n song_id UUID,\n duration_played INT,\n context_type TEXT,\n context_id UUID,\n PRIMARY KEY (user_id, listened_at)\n) WITH CLUSTERING ORDER BY (listened_at DESC);\n```\n\n\nWith this schema, we can efficiently query \"last 50 songs user X played\" since data is partitioned by user and ordered by timestamp.\n\n**Why Cassandra handles the write volume:** With 200 million DAU and an average of 5 songs per session, we are writing 1 billion events per day. Cassandra distributes this across nodes and handles writes without locks, achieving the throughput we need.\n\n---\n\n# 6. Design Deep Dive\n\nThe high-level architecture gives us a solid foundation, but system design interviews often go deeper into specific components. In this section, we will explore the trickiest parts of our design: audio streaming architecture, search infrastructure, personalization, scaling strategies, and offline playback.\n\nThese are the topics that distinguish a good system design answer from a great one.\n\n## 6.1 Audio Streaming Architecture\n\nDelivering audio with low latency and high quality is the core technical challenge. A user clicks play and expects music within 200ms. Once playing, there should be no buffering even when network conditions change. Let's explore how to achieve this at scale.\n\n### Audio File Preparation\n\nBefore a song can be streamed, it goes through an ingestion pipeline that prepares it for delivery:\n\n\n```mermaid\nflowchart LR\n A[Original Audio
WAV/FLAC]:::rose --> B[Transcoding
Service]:::primary\n B --> C[Quality Variants]:::orange\n B --> D[Chunking]:::orange\n B --> E[Encryption
DRM]:::orange\n C & D & E --> F[(Object Storage)]:::teal\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n#### Step 1: Transcoding\n\nThe original high-quality audio (often WAV or FLAC from the label) is converted to multiple quality levels:\n\n\n| Quality | Bitrate | File Size (4 min song) | Target Users |\n|---------|---------|------------------------|--------------|\n| Very High | 320 kbps | ~10 MB | Premium users on WiFi |\n| High | 160 kbps | ~5 MB | Standard quality |\n| Normal | 96 kbps | ~3 MB | Low bandwidth/data saver |\n| Low | 24 kbps | ~720 KB | Extreme data saver |\n\n\nMultiple quality levels enable adaptive streaming: if network conditions degrade, the client can switch to a lower quality mid-song rather than buffering.\n\n#### Step 2: Chunking\n\nEach quality variant is split into small chunks, typically 5-10 seconds each. A 4-minute song becomes roughly 24-48 chunks per quality level.\n\nWhy chunk? Several reasons:\n\n- **Faster start:** The client only needs the first chunk to start playing, not the entire file\n- **Efficient seeking:** Jumping to 2:30 means fetching only that chunk, not everything before it\n- **Quality switching:** The client can switch quality on chunk boundaries without audible artifacts\n- **Better caching:** Individual chunks can be cached and evicted independently\n\n#### Step 3: DRM Encryption\n\nEach chunk is encrypted for digital rights management. The client decrypts on playback using a license key tied to the user's subscription. This prevents users from extracting and redistributing audio files.\n\n### Streaming Protocol Options\n\nThere are several ways to deliver audio to clients. Let's compare the approaches:\n\n#### Approach 1: Progressive Download\n\nThe simplest approach: the client requests the complete audio file and downloads it sequentially.\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant CDN\n\n Client->>CDN: GET /song_123.mp3\n CDN-->>Client: Stream entire file (5 MB)\n Note over Client: Plays as it downloads\n```\n\n\n\n### Pros\n\n- Simple to implement with standard HTTP\n- Works everywhere\n\n### Cons\n\n- Cannot adapt quality mid-stream\n- Seeking to the middle requires downloading everything before it (or range requests)\n- No graceful handling of network changes\n\n\n#### Approach 2: HTTP Live Streaming (HLS) or DASH\n\nThe modern standard for adaptive streaming. Audio is pre-chunked, and a manifest file describes available qualities and chunk URLs.\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant CDN\n\n Client->>CDN: GET /song_123/manifest.m3u8\n CDN-->>Client: Return manifest (quality levels + chunk URLs)\n Note over Client: Choose 320kbps based on bandwidth\n Client->>CDN: GET /song_123/320/chunk_001.ts\n CDN-->>Client: First chunk\n Note over Client: Start playback immediately\n Client->>CDN: GET /song_123/320/chunk_002.ts\n Note over Client: Network slows down\n Client->>CDN: GET /song_123/160/chunk_003.ts\n Note over Client: Switch to 160kbps seamlessly\n```\n\n\n**How it works:**\n\n1. Client fetches the manifest file listing all available quality levels and chunk URLs\n2. Based on measured bandwidth, client selects a quality level\n3. Client downloads chunks sequentially, measuring download speed\n4. If bandwidth changes, client switches quality at the next chunk boundary\n\n\n### Pros\n\n- Adapts to changing network conditions\n- Efficient seeking (jump to any chunk)\n- Works well with CDN caching (chunks are small, static files)\n- Industry standard with wide client support\n\n### Cons\n\n- More complex than progressive download\n- Slight overhead from manifest requests\n\n\n#### Approach 3: Custom Protocol\n\nSpotify uses a proprietary streaming protocol optimized specifically for music. Key optimizations include:\n\n- **Predictive buffering:** Pre-fetches the next song while the current one is playing\n- **Gapless playback:** No silence between tracks in an album\n- **Multiple connections:** Uses parallel connections for redundancy\n- **Stutter-free algorithms:** Prioritizes playback continuity over quality\n\n#### Recommendation for interviews\n\nUse HLS/DASH as your answer. It's well understood, handles the core requirements (adaptive streaming, CDN compatibility), and is what most new streaming services would choose. Mention that mature services like Spotify have evolved custom protocols for specialized optimizations.\n\n### CDN Caching Strategy\n\nWith 5 PB of daily bandwidth, CDN caching is essential for both cost and latency. But not all songs are equal: some are played millions of times per day, while others might be played once per year.\n\n\n```mermaid\nflowchart TB\n subgraph \"Tiered Caching Strategy\"\n Edge[Edge Cache
Top 1% songs
~99% of traffic]:::green\n Regional[Regional Cache
Top 10% songs
Remaining hot content]:::orange\n Origin[Origin - S3
All 100M songs
Long tail content]:::teal\n end\n\n Edge -.->|miss| Regional\n Regional -.->|miss| Origin\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### The Pareto Effect in Music\n\nMusic consumption follows an extreme power law. The top 1% of songs account for roughly 90% of all streams, the top 10% covers more than 99% of streams, and the remaining 90% of the catalog, the \"long tail,\" is rarely accessed.\n\nThis distribution makes caching highly effective. Taylor Swift's latest single is requested millions of times per day and should be cached at every edge location. An obscure jazz recording from the 1960s might be requested once per month and can stay at origin.\n\n#### Cache Warming Strategies\n\n1. **New releases:** When an anticipated album drops (Beyonce, Taylor Swift), pre-push it to all edge locations before the release time\n2. **Trending content:** Monitor stream velocity and proactively cache songs that are gaining momentum\n3. **Predictive caching:** When a user starts a playlist, pre-cache upcoming songs at their nearest edge\n4. **Geographic patterns:** Songs popular in specific regions (K-pop in Korea, Reggaeton in Latin America) are prioritized at regional caches\n\n#### Cache Headers\n\nFor audio chunks, we set long cache times since content never changes:\n\n\n```shell\nCache-Control: public, max-age=31536000, immutable\n```\n\n\nThe `immutable` directive tells browsers and CDNs that the content will never change, eliminating conditional requests.\n\n---\n\n## 6.2 Search and Discovery\n\nSearch is how users find music. They type a few letters and expect to see relevant results instantly, even if they misspell the artist name or only remember part of the song title. Search must be fast (under 100ms), forgiving of errors, and smart about ranking.\n\n### Search Index Architecture\n\nWe do not query the primary database for search. Instead, we maintain a dedicated search index using Elasticsearch (or OpenSearch). This separation allows us to optimize each system for its specific workload.\n\n\n```mermaid\nflowchart LR\n subgraph \"Data Flow\"\n DB[(PostgreSQL
Source of Truth)]:::teal --> Sync[Change Data
Capture]:::secondary\n Sync --> ES[(Elasticsearch
Search Index)]:::primary\n end\n\n subgraph \"Query Flow\"\n User[User Query]:::rose --> SS[Search Service]:::primary\n SS --> ES\n ES --> SS\n SS --> Results[Ranked Results]:::green\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n#### What Gets Indexed\n\nThe Elasticsearch index holds a denormalized view of the catalog, pulling fields from several source tables so that a single query can match songs by title, artist, album name, lyrics, or genre without joins.\n\n\n| Field | Source | Notes |\n|-------|--------|-------|\n| Song titles | Song metadata | Primary search target |\n| Artist names | Artist table | Includes aliases and collaborations |\n| Album names | Album table | Including compilation names |\n| Lyrics | Lyrics service | For \"search by lyric\" feature |\n| Genres and moods | Tagging system | \"upbeat pop\", \"chill electronic\" |\n\n\n#### Index Structure\n\nElasticsearch uses inverted indexes, mapping terms to the documents containing them:\n\n\n```shell\n\"bohemian\" -> [song_123, song_456, album_789]\n\"queen\" -> [artist_001, song_123, song_456, album_789]\n\"rhapsody\" -> [song_123, song_888]\n```\n\n\nWhen a user searches \"bohemian rhapsody\", Elasticsearch finds documents matching both terms and ranks them by relevance.\n\n### Handling Typos and Fuzzy Matching\n\nUsers frequently misspell artist and song names. \"Ariana Grandi\", \"Tylor Swift\", \"bohemain rapsody\" should all return correct results.\n\nElasticsearch provides several mechanisms:\n\n1. **Fuzzy queries:** Allow for character edits (insertions, deletions, substitutions). \"bohemain\" with edit distance 1 matches \"bohemian\".\n2. **Phonetic matching:** \"Cue\" sounds like \"Queue\". Phonetic analyzers (Soundex, Metaphone) index words by how they sound.\n3. **N-gram tokenization:** Index partial words for autocomplete. \"bohe\" matches \"bohemian\" because we index [\"b\", \"bo\", \"boh\", \"bohe\", \"bohem\", ...].\n\n### Search Ranking\n\nRaw text matching is not enough. When searching \"queen\", users probably want the legendary rock band, not every song with \"queen\" in the title. We rank results using multiple signals:\n\n\n```mermaid\nflowchart TB\n subgraph \"Ranking Signals\"\n Text[Text Relevance
How well query matches]:::primary\n Pop[Popularity
Stream count, listeners]:::orange\n Pers[Personalization
User's listening history]:::green\n Fresh[Recency
New releases boost]:::teal\n end\n\n Text --> Score[Final
Score]:::primary\n Pop --> Score\n Pers --> Score\n Fresh --> Score\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n\n| Signal | Weight | Description |\n|--------|--------|-------------|\n| Text relevance | High | Exact matches rank higher than partial matches |\n| Popularity | High | Monthly listeners, total streams, trending velocity |\n| Personalization | Medium | Boost artists the user has listened to before |\n| Recency | Low | New releases get a small boost |\n\n\nThe weights are tuned based on user behavior data. If users consistently click the 3rd result, something is wrong with the ranking.\n\n### Autocomplete\n\nAs users type, we show suggestions in real-time. This requires extremely low latency (under 50ms) because users are actively typing and waiting.\n\n\n```mermaid\nsequenceDiagram\n participant User\n participant Client\n participant Search as Search Service\n participant ES as Elasticsearch\n\n User->>Client: Types \"tay\"\n Note over Client: Debounce 100ms\n Client->>Search: GET /autocomplete?q=tay\n Search->>ES: Prefix query with edge n-grams\n ES-->>Search: [Taylor Swift, Tay Keith, ...]\n Search-->>Client: Top 5 suggestions\n Client-->>User: Show dropdown\n Note over User: Total: ~50ms\n```\n\n\n#### Implementation\n\n1. Index names with edge n-grams: \"taylor\" becomes [\"t\", \"ta\", \"tay\", \"tayl\", \"taylo\", \"taylor\"]\n2. Use Elasticsearch's completion suggester, optimized for prefix matching\n3. Return top 5-10 suggestions sorted by popularity\n4. Client-side debouncing (100ms) prevents excessive requests while typing\n\n---\n\n## 6.3 Personalization and Recommendations\n\nPersonalization is what makes Spotify addictive. When you open the app and \"Discover Weekly\" has exactly the kind of music you love but have never heard, it feels like magic. That magic is the result of sophisticated recommendation systems running at massive scale.\n\n### Understanding User Taste\n\nThe recommendation system learns user preferences from multiple signals, each providing different information:\n\n\n```mermaid\nflowchart TB\n subgraph \"Explicit Signals (Strong)\"\n Like[Liked Songs
User explicitly saved]:::green\n Playlist[Added to Playlist
Deliberate action]:::green\n end\n\n subgraph \"Implicit Signals (Volume)\"\n History[Listening History
What they play]:::primary\n Skip[Skip Behavior
Skipped = dislike]:::orange\n Duration[Time Spent
Full song vs partial]:::primary\n end\n\n subgraph \"Context Signals\"\n Time[Time of Day
Morning vs night]:::secondary\n Device[Device Type
Phone, speaker, car]:::secondary\n Activity[Activity
Workout, focus, sleep]:::secondary\n end\n\n Like & Playlist & History & Skip & Duration & Time & Device & Activity --> Model[User Taste
Model]:::teal\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n\n| Signal | Type | Strength | Notes |\n|--------|------|----------|-------|\n| Liked songs | Explicit | Very strong | User clicked heart, clear preference |\n| Added to playlist | Explicit | Strong | Deliberate curation |\n| Listening history | Implicit | Medium | What they play repeatedly |\n| Skip behavior | Implicit | Medium | Skipping within 30 seconds indicates dislike |\n| Listen duration | Implicit | Weak | Full song = engagement, partial = maybe skip |\n| Time of day | Context | Weak | Morning playlists differ from night |\n\n\n### Recommendation Approaches\n\nThere are several approaches to generating recommendations, each with trade-offs:\n\n#### Approach 1: Collaborative Filtering\n\nThe core idea: users with similar taste like similar songs. If you and I both love the same 50 songs, and you love a song I have not heard, I will probably like it too.\n\n**How it works:**\n\n\n```mermaid\nflowchart LR\n subgraph \"User-Song Matrix\"\n M[Millions of users
× Millions of songs]:::rose\n end\n\n subgraph \"Matrix Factorization\"\n U[User Embeddings
100-dim vectors]:::primary\n S[Song Embeddings
100-dim vectors]:::primary\n end\n\n M --> |ALS/SVD| U\n M --> |ALS/SVD| S\n U --> |dot product| P[Predicted
Preferences]:::green\n S --> P\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n1. Build a sparse matrix of user-song interactions (plays, likes, skips)\n2. Use matrix factorization (ALS, SVD) to find latent factors\n3. Each user and song becomes a vector in latent space\n4. Predict preference as the dot product of user and song vectors\n\n\n### Pros\n\n- Discovers unexpected recommendations (\"users like you also like...\")\n- No need for content analysis\n\n### Cons\n\n- Cold start problem: new users and new songs have no interaction data\n- Computationally expensive: the matrix has billions of entries\n\n\n#### Approach 2: Content-Based Filtering\n\nThe idea: recommend songs similar to what the user already likes, based on the songs' characteristics.\n\n**How it works:**\n\n1. Extract audio features from songs: tempo (BPM), energy, acousticness, danceability, valence (happiness)\n2. Convert each song into an embedding (vector representation)\n3. Find songs with similar embeddings to what the user likes\n\n\n### Pros\n\n- Works for new songs (they have audio features even without play data)\n- Explainable: \"we recommend this because it has similar energy to songs you like\"\n\n### Cons\n\n- Can create filter bubbles (only recommends similar-sounding music)\n- Requires audio analysis infrastructure\n\n\n#### Approach 3: Hybrid (What Spotify Uses)\n\nReal systems combine multiple approaches:\n\n\n```mermaid\nflowchart TB\n subgraph \"Input Signals\"\n CF[Collaborative
Filtering]:::primary\n CB[Content-Based
Filtering]:::orange\n Context[Context
Model]:::secondary\n end\n\n subgraph \"Ensemble\"\n Combine[Ensemble Layer
Weighted combination]:::teal\n end\n\n subgraph \"Output\"\n Recs[Personalized
Recommendations]:::green\n end\n\n CF --> Combine\n CB --> Combine\n Context --> Combine\n Combine --> Recs\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe ensemble layer learns how to weight each model's predictions. For new users, content-based gets more weight. For users with rich history, collaborative filtering dominates.\n\n### Batch vs Real-Time Recommendations\n\nDifferent recommendation features have different latency requirements:\n\n\n| Feature | Compute Time | Refresh Rate | Infrastructure |\n|---------|--------------|--------------|----------------|\n| Discover Weekly | Hours | Weekly | Spark batch jobs |\n| Daily Mix | Hours | Daily | Spark batch jobs |\n| \"Because you listened to...\" | <100ms | Per request | Model serving + feature store |\n| Radio (similar to current song) | <100ms | Per request | Nearest neighbor search |\n\n\n**Batch recommendations** (Discover Weekly) run offline using full model training. They are pre-computed for every user and stored in Redis for instant serving.\n\n**Real-time recommendations** use pre-trained models with real-time features. When you finish a song, we query the model with your current session context to suggest the next song.\n\n### Recommendation System Architecture\n\nThe full pipeline connects event ingestion through Kafka, stream and batch processing, model training, and the serving layer that delivers recommendations to the API. Each stage is decoupled so that a slow batch job does not delay real-time serving.\n\n\n```mermaid\nflowchart TB\n subgraph \"Data Collection\"\n Events[User Events
plays, skips, likes]:::rose --> Kafka[Kafka]:::secondary\n end\n\n subgraph \"Processing\"\n Kafka --> Stream[Stream Processing
Flink/Spark Streaming]:::primary\n Stream --> Features[(Feature Store)]:::teal\n Batch[Batch Pipeline
Spark]:::green --> Features\n Batch --> Models[Train Models]:::green\n end\n\n subgraph \"Serving\"\n Models --> ModelServing[Model Serving
TensorFlow Serving]:::orange\n Features --> ModelServing\n ModelServing --> API[Recommendation API]:::primary\n end\n\n subgraph \"Storage\"\n API --> Cache[(Redis
Pre-computed recs)]:::orange\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n1. **Event collection:** Every play, skip, and like flows through Kafka\n2. **Stream processing:** Real-time features are computed (current session, recent plays)\n3. **Batch processing:** Heavy model training runs on Spark, retraining weekly\n4. **Feature store:** Central repository for all user and song features\n5. **Model serving:** Trained models are deployed for real-time inference\n6. **Caching:** Pre-computed recommendations (Discover Weekly) are stored in Redis\n\n---\n\n## 6.4 Handling Scale and High Availability\n\nWith 200 million daily active users and 1 billion streams per day, the system must be designed for extreme scale. Equally important, it must stay up. Users expect music to be available 24/7, and outages are highly visible on social media.\n\n### Horizontal Scaling\n\nThe key to horizontal scaling is keeping services stateless. When a service stores no state locally, you can add or remove instances without coordination.\n\n\n```mermaid\nflowchart TB\n LB[Load Balancer]:::secondary --> PS1[Playback Service 1]:::primary\n LB --> PS2[Playback Service 2]:::primary\n LB --> PS3[Playback Service 3]:::primary\n LB --> PS4[Playback Service N...]:::primary\n\n PS1 & PS2 & PS3 & PS4 --> Redis[(Redis Cluster)]:::orange\n PS1 & PS2 & PS3 & PS4 --> DB[(Databases)]:::teal\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n#### All our application services are stateless\n\nThe Playback Service can handle any request on any instance, the Search Service queries Elasticsearch, the Playlist Service reads from and writes to Cassandra, and the Recommendation Service fetches from the cache or feature store.\n\n#### Auto-scaling with Kubernetes\n\nWe use Horizontal Pod Autoscaler (HPA) to automatically add capacity:\n\n\n| Metric | Target | Action |\n|--------|--------|--------|\n| CPU utilization | 70% | Add pods when exceeded |\n| Request latency (p99) | 100ms | Add pods if latency spikes |\n| Request queue depth | 100 | Add pods if queue builds up |\n\n\nDuring peak hours (evening commute, weekends), the system automatically scales up. During low-traffic periods (3 AM), it scales down to save costs.\n\n### Database Scaling\n\nDifferent databases scale differently:\n\n#### Cassandra (Playlists, Listening History)\n\nCassandra scales horizontally by adding nodes. Data is automatically rebalanced across the cluster using consistent hashing.\n\n\n```mermaid\nflowchart TB\n subgraph \"Cassandra Cluster\"\n N1[Node 1
Users A-F]:::primary\n N2[Node 2
Users G-M]:::primary\n N3[Node 3
Users N-S]:::primary\n N4[Node 4
Users T-Z]:::primary\n end\n\n Service[Playlist Service]:::secondary --> N1 & N2 & N3 & N4\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nEach user's data lives on a subset of nodes (replication factor = 3). Reads and writes for a user hit the same nodes, making operations efficient.\n\n#### PostgreSQL (Song Metadata)\n\nFor the song catalog, we use read replicas rather than sharding. One primary handles writes such as song additions and metadata updates, while multiple read replicas handle read traffic.\n\nThis works because writes are infrequent, since new songs are added daily rather than per second, and 100 million songs fits comfortably in a well-indexed PostgreSQL instance.\n\n#### Elasticsearch (Search)\n\nElasticsearch scales by adding nodes to the cluster. Indexes are automatically sharded across nodes.\n\n### Multi-Layer Caching\n\nCaching is essential at every layer to handle our traffic:\n\n\n```mermaid\nflowchart LR\n Client[Client]:::rose --> ClientCache[Client Cache
Local Storage]:::green\n ClientCache --> CDN[CDN Cache
Edge Locations]:::green\n CDN --> AppCache[Redis Cache
Data Center]:::orange\n AppCache --> DB[(Database)]:::teal\n\n classDef rose fill:#f783ac,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n\n| Layer | What's Cached | TTL | Hit Rate |\n|-------|---------------|-----|----------|\n| Client | Recently played, user preferences | 1 hour | Very high |\n| CDN | Audio files, album art, static assets | 24+ hours | 80-90% |\n| Redis | Song metadata, recommendations, sessions | 5-15 min | 90%+ |\n| Database | Query results in buffer pool | Automatic | 95%+ |\n\n\nWith these cache layers, only a tiny fraction of requests reach the primary databases.\n\n### High Availability\n\n#### Multi-Region Deployment\n\nWe deploy in multiple geographic regions with active-active configuration:\n\n\n```mermaid\nflowchart TB\n subgraph \"US East\"\n US_LB[Load Balancer]:::secondary\n US_Services[Services]:::primary\n US_DB[(Databases)]:::teal\n end\n\n subgraph \"Europe\"\n EU_LB[Load Balancer]:::secondary\n EU_Services[Services]:::primary\n EU_DB[(Databases)]:::teal\n end\n\n subgraph \"Asia\"\n ASIA_LB[Load Balancer]:::secondary\n ASIA_Services[Services]:::primary\n ASIA_DB[(Databases)]:::teal\n end\n\n DNS[DNS/Global
Load Balancer]:::orange --> US_LB\n DNS --> EU_LB\n DNS --> ASIA_LB\n\n US_DB <-.->|replication| EU_DB\n EU_DB <-.->|replication| ASIA_DB\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nUsers are routed to the nearest region via DNS-based load balancing, and each region can handle its traffic independently. If one region fails, traffic is automatically routed to the remaining regions.\n\nData is replicated across regions, eventually consistent for playlists and synchronous for critical user data.\n\n#### Graceful Degradation\n\nWhen components fail, the system should degrade gracefully rather than crash entirely:\n\n\n| Component | If It Fails | Fallback |\n|-----------|-------------|----------|\n| Recommendation Service | Home page shows generic playlists | Serve cached \"top hits\" |\n| Search Service | Search is temporarily unavailable | Show recently played |\n| Playlist Service | Cannot modify playlists | Read from cache (read-only mode) |\n| CDN | Slower audio delivery | Fall back to origin |\n\n\nThe most important invariant: **playback should always work**. Users can tolerate degraded recommendations, but not being able to play music is a critical failure.\n\n### Rate Limiting\n\nProtection against abuse and accidental overload:\n\n\n| Endpoint | Limit | Notes |\n|----------|-------|-------|\n| Stream requests | 100/min per user | Prevents bot abuse |\n| Search requests | 30/min per user | Prevents scraping |\n| Playlist writes | 60/min per user | Prevents spam |\n| API (unauthenticated) | 10/min per IP | Prevents anonymous abuse |\n\n\nRate limits are enforced at the API Gateway using Redis-backed counters with sliding window algorithm.\n\n---\n\n# 7. Follow-ups\n\n## 7.1 Offline Playback\n\nPremium users can download songs for offline listening. This is essential for users on flights, subways, or in areas with poor connectivity. But it introduces unique challenges: we are giving users permanent copies of copyrighted content that we need to protect and eventually revoke.\n\n### Download Architecture\n\nThe download flow is similar to streaming, but the client stores the content locally instead of playing it immediately:\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant API as Download Service\n participant License as License Server\n participant CDN\n\n Client->>API: Request download for playlist\n Note over API: Verify premium subscription,
region, device count\n API->>License: Generate download licenses\n License-->>API: Return encrypted license keys\n API-->>Client: Download URLs + license keys\n\n loop For each song\n Client->>CDN: Download encrypted audio\n CDN-->>Client: Encrypted audio file\n Note over Client: Store locally with license\n end\n\n Note over Client: Later, offline playback\n Client->>Client: Decrypt using cached license\n Note over Client: Check license expiry\n```\n\n\n### The Download Flow in Detail\n\n1. **User initiates download:** User marks a playlist for offline access\n2. **Entitlement check:** Download Service verifies:\n - User has an active premium subscription\n - Song is available for download in user's region\n - User has not exceeded device limit (typically 5 devices)\n - User has not exceeded download limit (typically 10,000 songs)\n3. **License generation:** For each song, we generate an encrypted license tied to:\n - The user's account\n - The specific device\n - An expiration time (30 days)\n4. **Download:** Client downloads encrypted audio files from CDN\n5. **Local storage:** Audio and licenses are stored in encrypted local storage\n\n### DRM and Content Protection\n\nDownloaded files must be protected from piracy. If users could extract and share downloaded MP3s, the entire licensing model would collapse.\n\n\n```mermaid\nflowchart LR\n subgraph \"Protection Layers\"\n Encrypt[Audio Encryption
Widevine/FairPlay]:::primary\n License[License Keys
User + Device bound]:::orange\n Check[Periodic License Check
Every 30 days]:::green\n Limits[Device Limits
Max 5 devices]:::teal\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### Encryption\n\nAudio files are encrypted using industry-standard DRM systems: Widevine on Android and Chrome, FairPlay on iOS and Safari, and PlayReady on Windows. The decryption keys are stored in secure hardware (when available) and never exposed to the application layer.\n\n#### License Expiration\n\nDownloaded content does not last forever. Licenses expire after 30 days of offline use. When the device comes online, it must check with the License Server to renew. If the user's subscription has lapsed, renewal fails and downloaded content becomes unplayable.\n\n#### Device Management\n\n\n| Limit | Value | Reason |\n|-------|-------|--------|\n| Max devices per account | 5 | Prevent account sharing abuse |\n| Max downloads per device | 10,000 songs | Storage sanity |\n| Offline validity | 30 days | Periodic subscription verification |\n\n\n### Storage Management on Device\n\nThe client is responsible for managing downloaded content:\n\n- **Quality selection:** Users choose download quality (affects storage)\n - Very High (320 kbps): ~10 MB per song\n - High (160 kbps): ~5 MB per song\n - Normal (96 kbps): ~3 MB per song\n- **Storage tracking:** Show users how much storage is used\n- **Smart cleanup:** Automatically remove songs that have not been played offline in 60+ days, oldest downloads first when storage is low\n\n---\n\n## 7.2 Royalty Calculation and Playback Tracking\n\nEvery stream must be tracked accurately for royalty payments to artists and labels. This is not only a feature; it's a legal and financial requirement. Billions of dollars flow from Spotify to rights holders based on stream counts.\n\n### What Counts as a Stream?\n\nNot every play counts as a stream for royalty purposes:\n\n\n| Condition | Counts as Stream? | Why |\n|-----------|-------------------|-----|\n| Played 30+ seconds | Yes | Industry standard threshold |\n| Played < 30 seconds | No | Prevents gaming with short plays |\n| Automated/bot playback | No | Detected and filtered |\n| User actively listening | Yes | Normal usage |\n| Background audio with no engagement | Maybe | Complex rules apply |\n\n\nThe 30-second threshold is an industry standard. Playing 29 seconds and skipping does not count. Playing 30 seconds and then skipping does count.\n\n### Playback Event Pipeline\n\nWe need to capture every play event, process it at scale, and aggregate for royalty calculation:\n\n\n```mermaid\nflowchart TB\n subgraph \"Event Collection\"\n Client[Client Apps]:::rose --> Events[Playback Events
start, pause, seek, complete]:::primary\n Events --> Collector[Event Collector
API]:::primary\n end\n\n subgraph \"Stream Processing\"\n Collector --> Kafka[(Kafka
Durable queue)]:::orange\n Kafka --> Processor[Stream Processor
Flink/Spark Streaming]:::primary\n Processor --> Fraud[Fraud Detection
ML Models]:::red\n end\n\n subgraph \"Storage & Analytics\"\n Processor --> Analytics[(Analytics DB
ClickHouse)]:::teal\n Processor --> Royalty[Royalty
Aggregator]:::green\n Royalty --> Payments[(Payment
System)]:::green\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n### Event Flow in Detail\n\n1. **Client emits events:** The app sends events for key playback moments:\n - `play_start`: User pressed play\n - `play_pause`: User paused\n - `play_seek`: User jumped to different position\n - `play_complete`: Song finished naturally\n - `play_skip`: User skipped before completion\n2. **Event collection:** Events are batched and sent to our Event Collector API. We use batching to reduce network overhead and handle intermittent connectivity.\n3. **Kafka for durability:** Events flow into Kafka, providing durability and buffering. If downstream systems are slow, Kafka absorbs the backlog without losing data.\n4. **Stream processing:** A Flink or Spark Streaming job processes events in near-real-time:\n - Reconstructs complete play sessions from individual events\n - Determines which plays qualify as streams (30+ seconds)\n - Enriches with user and song metadata\n5. **Fraud detection:** ML models flag suspicious patterns before counting streams\n6. **Analytics and royalties:** Valid streams are written to analytics databases and fed into the royalty calculation pipeline\n\n### Fraud Detection\n\nStream manipulation is a real problem. Bad actors try to inflate stream counts to earn royalties fraudulently. We detect and filter these:\n\n\n```mermaid\nflowchart LR\n subgraph \"Fraud Signals\"\n Bot[Bot-like Patterns
Same song on loop]:::red\n Geo[Geographic Anomalies
Impossible location changes]:::red\n Device[Device Fingerprinting
Emulators, VMs]:::red\n Behavior[Account Behavior
No variety, no playlists]:::red\n end\n\n Bot & Geo & Device & Behavior --> ML[ML Fraud
Classifier]:::primary\n ML --> Decision{Fraud Score}:::yellow\n Decision -->|High| Block[Block + Investigate]:::red\n Decision -->|Low| Count[Count as Stream]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n```\n\n\n#### Detection signals\n\nThe fraud classifier draws on five categories of signal, ranging from obvious replay loops to subtle geographic and device anomalies that distinguish legitimate listeners from automated accounts.\n\n\n| Signal | Description | Indicates |\n|--------|-------------|-----------|\n| Loop behavior | Same song played 100+ times | Bot or manipulation |\n| Geographic impossibility | User in Tokyo, then NYC 5 minutes later | Account compromise |\n| Device fingerprint | Virtual machine, emulator, automation | Bot farm |\n| No diversity | Only plays one artist, no browsing | Targeted manipulation |\n| Payment anomalies | Many streams but subscription always fails | Stolen credentials |\n\n\nDetected fraud is flagged, excluded from royalty calculations, and investigated. Repeat offenders have accounts suspended.\n\n### Royalty Calculation\n\nStream data feeds into the royalty system that determines artist payments. The calculation is complex and varies by contract, but the basic model is:\n\n1. **Pool model:** Total subscription revenue forms a pool\n2. **Stream share:** Each artist's streams as a percentage of total streams\n3. **Payment:** Artist receives their percentage of the revenue pool\n\nThis happens monthly, processing billions of stream records to determine millions of payments.\n\n---\n\n# Quiz","pageType":"system-design-interviews"}

Get Premium