{"title":"Design Discord","description":"","content":"\n> **QUESTION**\n>\n> #### What is Discord?\n>\n> Discord is a real-time communication platform that combines text messaging, voice chat, and video calling within organized communities called \"servers.\" Each server contains multiple channels for different topics, allowing communities to segment conversations while keeping everything in one place.\n>\n> \n> \n>\n> Originally built for gamers, Discord has expanded to support communities of all types, from study groups to professional teams to hobbyist clubs. The platform emphasizes low-latency communication, particularly for voice chat where even small delays disrupt natural conversation. Users can see who's online, what they're doing, and join voice channels with a single click.\n>\n> **Popular Examples:** Discord, Slack, Microsoft Teams, Telegram\n\n\nIn this chapter, we will explore the **high-level design of a Discord-like platform**.\n\nThis system design problem tests your ability to design real-time systems with WebSockets, handle presence at scale, implement voice/video infrastructure, and manage millions of concurrent connections.\n\nLet's start by clarifying the requirements:\n\n---\n\n# 1. Clarifying Requirements\n\nBefore diving into the design, it's important to ask thoughtful questions to uncover hidden assumptions, clarify ambiguities, and define the system's scope more precisely.\n\nHere is an example of how a discussion between the candidate and the interviewer might unfold:\n\n\n> **DISCUSSION**\n>\n> **Candidate:** \"What is the expected scale? How many concurrent users and messages should we support?\"\n>\n> **Interviewer:** \"Let's design for 10 million concurrent users, with about 1 billion messages sent per day. Peak concurrent users in a single server could be 500,000.\"\n>\n> **Candidate:** \"Should we support both text and voice/video communication?\"\n>\n> **Interviewer:** \"Yes, both are core features. Text messaging should be real-time with message history. Voice chat should support channels with hundreds of simultaneous participants.\"\n>\n> **Candidate:** \"How should messages be delivered? Do users need to see messages in real-time as they're sent?\"\n>\n> **Interviewer:** \"Yes, messages should appear instantly for all online users in a channel. We also need to support message history for when users come online later.\"\n>\n> **Candidate:** \"What about the presence system? Do users need to see who's online and their current status?\"\n>\n> **Interviewer:** \"Yes, users should see online/offline/idle/do-not-disturb status in real-time. They should also see what game or activity others are doing.\"\n>\n> **Candidate:** \"Should we support direct messages between users in addition to server channels?\"\n>\n> **Interviewer:** \"Yes, users can have private conversations with individuals or small groups outside of servers.\"\n>\n> **Candidate:** \"What are the latency requirements, especially for voice chat?\"\n>\n> **Interviewer:** \"Voice latency should be under 200ms for a good experience. Text messages should appear within 100ms for online users.\"\n\n\nAfter gathering the details, we can summarize the key system requirements.\n\n## 1.1 Functional Requirements\n\n- **Servers & Channels:** Users can create servers with multiple text and voice channels.\n- **Real-time Messaging:** Users can send and receive text messages instantly in channels.\n- **Message History:** Users can scroll back to view previous messages in a channel.\n- **Voice Chat:** Users can join voice channels and talk with others in real-time.\n- **Direct Messages:** Users can have private conversations with other users.\n- **Presence:** Users can see who's online and their current status/activity.\n\n#### Out of Scope:\n\n- **Video Streaming:** Screen sharing and video calls.\n- **File Storage:** Large file uploads and media galleries.\n- **Bots & Integrations:** Third-party bot framework.\n- **Moderation Tools:** Advanced moderation and automod features.\n\n## 1.2 Non-Functional Requirements\n\n- **High Availability:** The system must be highly available (99.99% uptime). Communication platforms are critical infrastructure.\n- **Low Latency:** Text messages delivered within 100ms. Voice latency under 200ms.\n- **Scalability:** Support 10M concurrent connections, 1B messages/day, voice channels with 500+ participants.\n- **Consistency:** Messages must be delivered in order and exactly once to each recipient.\n- **Global Reach:** Low-latency voice communication worldwide.\n\n---\n\n# 2. Back-of-the-Envelope Estimation\n\nTo understand the scale of our system, let's make some reasonable assumptions.\n\n#### **Messages (Writes)**\n\n- Messages per day: **1 billion**\n- Average message QPS = `1,000,000,000 / 86,400` ≈ **11,600 QPS (steady state)**\n- Peak message QPS (3x factor) ≈ **35,000 QPS**\n\n#### **Message Reads**\n\n- Each message is read by an average of **50 users** (channel members online)\n- Total message deliveries = `1B × 50` = **50 billion deliveries/day**\n- Average delivery QPS = `50,000,000,000 / 86,400` ≈ **580,000 QPS**\n- Peak delivery QPS ≈ **1.7 million QPS**\n\n#### **Concurrent Connections**\n\n- Concurrent users: **10 million**\n- Each user maintains **1 WebSocket connection**\n- Total concurrent WebSocket connections: **10 million**\n- Assuming each server handles 100K connections: **100 WebSocket servers** minimum\n\n#### **Voice Traffic**\n\n- Assume **5%** of concurrent users are in voice channels = **500,000 users**\n- Average voice channel size: **10 users**\n- Active voice channels: **50,000**\n- Audio bitrate per user: **64 kbps** (Opus codec)\n- Total voice bandwidth: `500K users × 64 kbps` ≈ **32 Gbps**\n\n#### **Storage**\n\n**Messages (per year):**\n\n- Messages per day: 1 billion\n- Average message size: 200 bytes (text + metadata)\n- Daily message storage = `1B × 200 bytes` = **200 GB/day**\n- Annual message storage ≈ **73 TB/year**\n\n**Message History Reads:**\n\n- Users frequently load message history (scrolling up)\n- Assume **10 billion** history fetches per day\n- This is read-heavy: optimize with caching\n\n---\n\n# 3. Core APIs\n\nDiscord needs APIs for real-time communication, server management, and presence. Most real-time features use WebSocket connections rather than REST APIs.\n\n### **1. WebSocket Gateway Connection**\n\n#### Connection: `wss://gateway.discord.com`\n\nThe primary interface for real-time events. Clients maintain a persistent WebSocket connection.\n\n##### **Connection Flow:**\n\n1. Client connects to gateway\n2. Server sends `HELLO` with heartbeat interval\n3. Client sends `IDENTIFY` with auth token\n4. Server sends `READY` with initial state\n5. Client/server exchange events bidirectionally\n\n##### **Event Types (Server → Client):**\n\n- **MESSAGE_CREATE**: New message in a channel\n- **MESSAGE_UPDATE**: Message edited\n- **MESSAGE_DELETE**: Message deleted\n- **PRESENCE_UPDATE**: User status changed\n- **VOICE_STATE_UPDATE**: User joined/left voice channel\n- **TYPING_START**: User started typing\n\n##### **Event Types (Client → Server):**\n\n- **HEARTBEAT**: Keep connection alive\n- **REQUEST_GUILD_MEMBERS**: Request member list\n- **UPDATE_PRESENCE**: Update own status\n- **VOICE_STATE_UPDATE**: Join/leave voice channel\n\n### **2. Send Message**\n\n#### Endpoint: `POST /api/v1/channels/{channel_id}/messages`\n\n##### **Request Parameters:**\n\n- **content** _(required)_: Message text (max 2000 characters).\n- **nonce** _(optional)_: Client-generated ID for deduplication.\n- **message_reference** _(optional)_: ID of message being replied to.\n\n##### **Sample Response:**\n\n- **message_id**: Unique snowflake ID.\n- **timestamp**: Server-assigned timestamp.\n- **author**: User object of sender.\n\n##### **Error Cases:**\n\n- `403 Forbidden`: User lacks permission to send in channel.\n- `429 Too Many Requests`: Rate limited.\n\n### **3. Get Message History**\n\n#### Endpoint: `GET /api/v1/channels/{channel_id}/messages`\n\n##### **Request Parameters:**\n\n- **before** _(optional)_: Get messages before this message ID.\n- **after** _(optional)_: Get messages after this message ID.\n- **limit** _(optional)_: Number of messages (default: 50, max: 100).\n\n##### **Sample Response:**\n\n- **messages**: Array of message objects, newest first.\n\n### **4. Join Voice Channel**\n\n#### Endpoint: `POST /api/v1/channels/{channel_id}/voice`\n\nThis initiates the voice connection flow.\n\n##### **Sample Response:**\n\n- **token**: Voice server authentication token.\n- **endpoint**: Voice server WebSocket URL.\n- **session_id**: Voice session identifier.\n\n##### **Flow:**\n\n1. Client calls this endpoint\n2. Receives voice server details\n3. Connects to voice WebSocket\n4. Establishes WebRTC connection for audio\n\n### **5. Update Presence**\n\n#### Endpoint: `PATCH /api/v1/users/@me/presence`\n\n##### **Request Parameters:**\n\n- **status** _(required)_: \"online\", \"idle\", \"dnd\", \"invisible\".\n- **activity** _(optional)_: Current activity (game, music, etc.).\n\n##### **Sample Response:**\n\n- **success**: Boolean indicating operation result.\n\n---\n\n# 4. High-Level Design\n\nAt a high level, our system must satisfy four core requirements:\n\n1. **Real-time Messaging:** Deliver text messages instantly to all channel members.\n2. **Message Persistence:** Store messages durably and allow history retrieval.\n3. **Voice Communication:** Enable low-latency voice chat in channels.\n4. **Presence System:** Track and broadcast user online status in real-time.\n\nThe system requires **persistent connections** (WebSockets) for real-time features and must handle **massive fanout** (one message to many recipients).\n\n**Note:** Instead of presenting the full architecture at once, we'll build it incrementally by addressing one requirement at a time. This approach is easier to follow and mirrors how you would explain the design in an interview.\n\n## 4.1 Requirement 1: Real-time Messaging\n\nWhen a user sends a message, it must be delivered instantly to all online members of that channel. This is the core real-time feature.\n\n### Components Needed\n\n#### **Gateway Service (WebSocket Servers)**\n\nMaintains persistent WebSocket connections with clients. This is the real-time backbone.\n\n**Responsibilities:**\n\n- Manage millions of concurrent WebSocket connections\n- Route incoming messages to appropriate handlers\n- Push events to connected clients\n- Handle heartbeats and connection lifecycle\n\n#### **Message Service**\n\nProcesses message creation, validation, and persistence.\n\n**Responsibilities:**\n\n- Validate message content and permissions\n- Assign message IDs and timestamps\n- Store messages durably\n- Publish messages for fanout\n\n#### **Pub/Sub System**\n\nEnables message fanout from sender to all recipients.\n\n**Responsibilities:**\n\n- Route messages to relevant Gateway servers\n- Support channel-based subscriptions\n- Handle high-throughput message distribution\n\n#### **Session Registry**\n\nTracks which users are connected to which Gateway servers.\n\n**Responsibilities:**\n\n- Map user IDs to Gateway server instances\n- Support efficient lookup for message routing\n- Handle user connection/disconnection\n\n### Flow: Sending a Message\n\n\n```mermaid\nsequenceDiagram\n participant Sender as Sender Client\n participant GW1 as Gateway Server 1\n participant Msg as Message Service\n participant DB as Database\n participant PubSub as Pub/Sub (Redis)\n participant GW2 as Gateway Server 2\n participant Recipient as Recipient Client\n\n Sender->>GW1: Send message (WebSocket)\n GW1->>Msg: Process message\n Msg->>Msg: Validate & assign ID\n Msg->>DB: Store message (async)\n Msg->>PubSub: Publish to channel topic\n PubSub->>GW1: Broadcast to subscribers\n PubSub->>GW2: Broadcast to subscribers\n GW1->>Sender: MESSAGE_CREATE event\n GW2->>Recipient: MESSAGE_CREATE event\n```\n\n\n1. Sender's client sends message via WebSocket to their **Gateway Server**.\n2. Gateway forwards to **Message Service** for processing.\n3. Message Service:\n\n- Validates content and user permissions\n- Assigns snowflake ID and timestamp\n- Stores message asynchronously to database\n\n1. Message Service publishes to **Pub/Sub** on the channel's topic.\n2. All Gateway servers subscribed to that channel receive the message.\n3. Each Gateway pushes `MESSAGE_CREATE` event to connected channel members.\n\n### Channel Subscriptions\n\nWhen a user connects:\n\n1. Gateway loads user's server and channel memberships\n2. Gateway subscribes to Pub/Sub topics for each channel\n3. When user switches channels, subscriptions update accordingly\n\n\n```shell\n# Pub/Sub Topic Structure\nchannel:{channel_id} # All messages for a channel\nguild:{guild_id} # Server-wide events\nuser:{user_id} # DMs and mentions for a user\n```\n\n\n---\n\n## 4.2 Requirement 2: Message Persistence\n\nMessages must be stored durably so users can view history when they join a channel or scroll up.\n\n### Components Needed\n\n#### **Message Database**\n\nStores all messages with efficient retrieval by channel and time.\n\n**Requirements:**\n\n- High write throughput (11K+ messages/second)\n- Efficient range queries (messages in channel, sorted by time)\n- Long-term retention (years of history)\n\n#### **Message Cache**\n\nCaches recent messages for fast access.\n\n**Responsibilities:**\n\n- Store last N messages per channel\n- Reduce database load for common queries\n- Handle \"load more\" for active channels\n\n### Flow: Loading Message History\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant API as REST API\n participant Cache as Redis Cache\n participant DB as Message DB\n\n Client->>API: GET /channels/{id}/messages?before=X\n API->>Cache: Check recent messages cache\n\n alt Cache Hit (recent messages)\n Cache-->>API: Return cached messages\n else Cache Miss\n API->>DB: Query messages before X\n DB-->>API: Return messages\n API->>Cache: Cache if recent\n end\n\n API-->>Client: Return messages array\n```\n\n\n1. Client requests message history via REST API (not WebSocket).\n2. API checks **Redis cache** for recent messages.\n3. On cache miss, queries **Message Database**.\n4. Returns messages sorted by timestamp (newest last).\n\n### Message ID Design (Snowflake)\n\nDiscord uses Twitter's Snowflake ID format for messages:\n\n\n```shell\n64-bit ID:\n┌────────────────────────────────────────────┬──────────┬──────────────┐\n│ Timestamp (42 bits) │Worker(10)│ Sequence(12) │\n│ Milliseconds since epoch │ Server ID│ Per-ms counter│\n└────────────────────────────────────────────┴──────────┴──────────────┘\n```\n\n\n**Benefits:**\n\n- **Time-sortable:** IDs are chronologically ordered\n- **Distributed:** No central coordination needed\n- **Efficient queries:** Range scans by ID = range scans by time\n- **High throughput:** 4096 IDs per millisecond per worker\n\n---\n\n## 4.3 Requirement 3: Voice Communication\n\nVoice chat requires ultra-low latency (< 200ms) and efficient bandwidth usage for channels with many participants.\n\n### Components Needed\n\n#### **Voice Gateway**\n\nCoordinates voice connections and manages voice state.\n\n**Responsibilities:**\n\n- Handle voice channel join/leave\n- Assign users to voice servers\n- Manage speaking state and muting\n\n#### **Voice Server (SFU)**\n\nSelective Forwarding Unit that routes audio streams between participants.\n\n**Responsibilities:**\n\n- Receive audio from each speaker\n- Forward audio to each listener\n- Handle audio mixing for large channels\n- Manage WebRTC connections\n\n#### **TURN/STUN Servers**\n\nHelp establish peer connections through NATs and firewalls.\n\n### Voice Architecture: SFU vs. Mesh vs. MCU\n\n\n| ##### Architecture | ##### Description | ##### Pros | ##### Cons | ##### Best For |\n| --- | --- | --- | --- | --- |\n| **Mesh** | Each user connects to every other user | Simple, no server | O(n²) connections | 2-4 users |\n| **SFU** | Server forwards streams without mixing | Scalable, flexible | Server bandwidth | 5-100+ users |\n| **MCU** | Server mixes all streams into one | Low client bandwidth | High server CPU | Large meetings |\n\n\n**Discord uses SFU (Selective Forwarding Unit):**\n\n- Each user sends one audio stream to server\n- Server forwards streams to each listener\n- Listeners receive separate streams (can choose quality)\n- Scales well for voice channels with many users\n\n### Flow: Joining Voice Channel\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant GW as Gateway\n participant VGW as Voice Gateway\n participant VS as Voice Server (SFU)\n participant Other as Other Clients\n\n Client->>GW: Voice State Update (join channel)\n GW->>VGW: Assign voice server\n VGW-->>GW: Voice server endpoint + token\n GW-->>Client: VOICE_SERVER_UPDATE event\n\n Client->>VS: Connect WebSocket\n VS-->>Client: Session description\n Client->>VS: WebRTC offer (SDP)\n VS-->>Client: WebRTC answer (SDP)\n Client->>VS: ICE candidates\n VS-->>Client: ICE candidates\n\n Note over Client,VS: WebRTC connection established\n\n Client->>VS: Send audio stream\n VS->>Other: Forward audio stream\n Other->>VS: Send audio streams\n VS->>Client: Forward audio streams\n```\n\n\n1. Client sends voice state update to join channel via **Gateway**.\n2. **Voice Gateway** assigns a nearby voice server.\n3. Client receives voice server endpoint and connects.\n4. Client and **Voice Server** exchange WebRTC signaling (SDP, ICE).\n5. WebRTC connection established for audio.\n6. Client sends audio to Voice Server.\n7. Voice Server selectively forwards to other participants.\n\n### Voice Server Selection\n\nFor low latency, users should connect to geographically close voice servers:\n\n\n```mermaid\nflowchart TB\n subgraph Users\n US_User[US User]\n EU_User[EU User]\n Asia_User[Asia User]\n end\n\n subgraph VoiceServers[\"Voice Servers\"]\n VS_US[Voice Server
US-East]\n VS_EU[Voice Server
EU-West]\n VS_Asia[Voice Server
Asia-Pacific]\n end\n\n US_User --> VS_US\n EU_User --> VS_EU\n Asia_User --> VS_Asia\n\n VS_US <--> VS_EU\n VS_EU <--> VS_Asia\n VS_US <--> VS_Asia\n\n style VS_US fill:#00ceff,stroke:#000,color:#000\n style VS_EU fill:#00ceff,stroke:#000,color:#000\n style VS_Asia fill:#00ceff,stroke:#000,color:#000\n```\n\n\n**For cross-region voice channels:**\n\n- Users connect to nearest voice server\n- Voice servers relay audio between regions\n- Adds latency for cross-region communication (unavoidable)\n\n---\n\n## 4.4 Requirement 4: Presence System\n\nUsers need to see who's online, their status, and current activity in real-time.\n\n### Components Needed\n\n#### **Presence Service**\n\nTracks and distributes user presence information.\n\n**Responsibilities:**\n\n- Track online/offline/idle/dnd status\n- Track current activity (game, music, etc.)\n- Handle presence subscriptions\n- Distribute presence updates\n\n### Presence Challenges\n\n**Scale of the Problem:**\n\n- 10 million concurrent users\n- Each user cares about presence of friends and server members\n- Average user has 100 friends + 500 server members they might see\n- A single user going offline could trigger 10,000+ notifications\n\n**Naive approach would explode:**\n\n- User A goes offline\n- Must notify everyone who has A as friend or shares a server\n- Could be millions of notifications for popular users\n\n### Presence Distribution Strategy\n\n**Lazy Presence Loading:**\n\nInstead of pushing all presence changes, use pull-with-subscription:\n\n1. When user opens a server/DM, fetch current presence of visible members\n2. Subscribe to presence updates only for visible members\n3. When user switches context, update subscriptions\n\n\n```mermaid\nflowchart LR\n subgraph Client[\"Client View\"]\n MemberList[Member List
50 visible users]\n end\n\n subgraph Presence[\"Presence Service\"]\n Cache[Presence Cache]\n Sub[Subscription Manager]\n end\n\n MemberList -->|1. Request presence| Cache\n Cache -->|2. Return current status| MemberList\n MemberList -->|3. Subscribe to updates| Sub\n Sub -->|4. Push changes| MemberList\n\n style Cache fill:#ff8787,stroke:#000,color:#000\n style Sub fill:#9775fa,stroke:#000,color:#000\n```\n\n\n**Presence Data Model:**\n\n\n```json\n{\n \"user_id\": \"123456789\",\n \"status\": \"online\",\n \"client_status\": {\n \"desktop\": \"online\",\n \"mobile\": \"idle\",\n \"web\": \"offline\"\n },\n \"activity\": {\n \"type\": \"playing\",\n \"name\": \"Valorant\",\n \"timestamps\": {\n \"start\": 1699900000000\n }\n },\n \"last_seen\": 1699903600000\n}\n```\n\n\n### Flow: Presence Update\n\n\n```mermaid\nsequenceDiagram\n participant Client\n participant GW as Gateway\n participant Presence as Presence Service\n participant Cache as Redis\n participant PubSub as Pub/Sub\n participant GW2 as Other Gateways\n participant Subscribers as Subscribed Clients\n\n Client->>GW: UPDATE_PRESENCE (status: idle)\n GW->>Presence: Update user presence\n Presence->>Cache: Store new presence\n Presence->>PubSub: Publish presence change\n PubSub->>GW2: Presence update\n GW2->>Subscribers: PRESENCE_UPDATE event\n```\n\n\n1. Client sends presence update (status change, activity change).\n2. **Gateway** forwards to **Presence Service**.\n3. Presence Service updates **Redis cache**.\n4. Publishes presence change to **Pub/Sub**.\n5. Gateways with subscribed users receive update.\n6. Push `PRESENCE_UPDATE` to subscribed clients.\n\n### Idle Detection\n\n**Server-side idle detection:**\n\n- Gateway tracks last activity timestamp per connection\n- If no activity for 5 minutes, mark user as idle\n- If WebSocket disconnects, mark user as offline after timeout\n\n**Client-side activity:**\n\n- Mouse movement, keyboard input\n- Application focus\n- Client sends heartbeat with activity flag\n\n---\n\n## 4.5 Putting It All Together\n\nAfter covering all requirements individually, here is the complete architecture:\n\n\n```mermaid\nflowchart TB\n subgraph Clients\n Desktop[Desktop Client]\n Mobile[Mobile Client]\n Web[Web Client]\n end\n\n subgraph Edge[\"Edge Layer\"]\n LB[Load Balancer]\n CDN[CDN
Static Assets]\n end\n\n subgraph Gateway[\"Gateway Layer\"]\n GW1[Gateway 1]\n GW2[Gateway 2]\n GWN[Gateway N]\n end\n\n subgraph Services[\"Application Services\"]\n Msg[Message Service]\n Presence[Presence Service]\n User[User Service]\n Guild[Guild Service]\n end\n\n subgraph Voice[\"Voice Infrastructure\"]\n VGW[Voice Gateway]\n VS1[Voice Server 1
SFU]\n VS2[Voice Server 2
SFU]\n end\n\n subgraph Realtime[\"Real-time Infrastructure\"]\n PubSub[Pub/Sub
Redis Cluster]\n Session[Session Registry
Redis]\n end\n\n subgraph Storage[\"Storage Layer\"]\n MsgDB[(Message DB
Cassandra)]\n MetaDB[(Metadata DB
PostgreSQL)]\n Cache[(Cache
Redis)]\n end\n\n Desktop --> LB\n Mobile --> LB\n Web --> CDN\n CDN --> LB\n\n LB --> GW1\n LB --> GW2\n LB --> GWN\n\n GW1 --> Msg\n GW1 --> Presence\n GW1 --> PubSub\n GW1 --> Session\n GW2 --> Msg\n GW2 --> PubSub\n GWN --> PubSub\n\n Msg --> MsgDB\n Msg --> PubSub\n Msg --> Cache\n\n Presence --> Cache\n Presence --> PubSub\n\n Guild --> MetaDB\n User --> MetaDB\n\n GW1 --> VGW\n VGW --> VS1\n VGW --> VS2\n\n style GW1 fill:#00ceff,stroke:#000,color:#000\n style GW2 fill:#00ceff,stroke:#000,color:#000\n style GWN fill:#00ceff,stroke:#000,color:#000\n style PubSub fill:#ff8787,stroke:#000,color:#000\n style Session fill:#ff8787,stroke:#000,color:#000\n style VS1 fill:#9775fa,stroke:#000,color:#000\n style VS2 fill:#9775fa,stroke:#000,color:#000\n style Msg fill:#38d9a9,stroke:#000,color:#000\n style Presence fill:#38d9a9,stroke:#000,color:#000\n style MsgDB fill:#69db7c,stroke:#000,color:#000\n style MetaDB fill:#69db7c,stroke:#000,color:#000\n style Cache fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n### Core Components Summary\n\n\n| ##### Component | ##### Purpose |\n| --- | --- |\n| Gateway Servers | Maintain WebSocket connections, route real-time events |\n| Message Service | Process, validate, and store messages |\n| Presence Service | Track and distribute user online status |\n| Voice Gateway | Coordinate voice channel connections |\n| Voice Servers (SFU) | Route audio streams between participants |\n| Pub/Sub (Redis) | Fan out messages and events to Gateway servers |\n| Session Registry | Track user-to-Gateway mapping |\n| Message DB (Cassandra) | Store message history |\n| Metadata DB (PostgreSQL) | Store users, servers, channels, permissions |\n| Cache (Redis) | Cache recent messages, presence, permissions |\n\n\n---\n\n# 5. Database Design\n\n### 5.1 SQL vs NoSQL\n\nOur system has different data access patterns that benefit from different database types:\n\n**Messages:**\n\n- Extremely high write volume (11K+ QPS)\n- Simple access pattern: messages by channel, sorted by time\n- Append-only (messages rarely updated)\n- **Choice: Cassandra** (optimized for time-series writes)\n\n**Servers, Channels, Users, Permissions:**\n\n- Complex relationships (users in servers, roles, permissions)\n- Read-heavy with occasional writes\n- Need for transactions (role updates, permission changes)\n- **Choice: PostgreSQL**\n\n**Presence, Sessions, Cache:**\n\n- Sub-millisecond reads required\n- Frequent updates\n- Ephemeral data\n- **Choice: Redis**\n\n### 5.2 Database Schema\n\n#### **1. Guilds (Servers) Table (PostgreSQL)**\n\nStores server metadata.\n\n\n| ##### Field | ##### Type | ##### Description |\n| --- | --- | --- |\n| `guild_id` | BIGINT (PK) | Snowflake ID |\n| `name` | VARCHAR(100) | Server name |\n| `owner_id` | BIGINT (FK) | Owner's user ID |\n| `icon_url` | TEXT | Server icon |\n| `member_count` | INTEGER | Number of members |\n| `created_at` | TIMESTAMP | Creation time |\n\n\n#### **2. Channels Table (PostgreSQL)**\n\nStores channel information.\n\n\n| ##### Field | ##### Type | ##### Description |\n| --- | --- | --- |\n| `channel_id` | BIGINT (PK) | Snowflake ID |\n| `guild_id` | BIGINT (FK) | Parent server (NULL for DMs) |\n| `type` | SMALLINT | 0=text, 2=voice, 1=DM, 3=group DM |\n| `name` | VARCHAR(100) | Channel name |\n| `topic` | TEXT | Channel description |\n| `position` | INTEGER | Display order |\n| `created_at` | TIMESTAMP | Creation time |\n\n\n**Indexes:**\n\n- `idx_channels_guild` on `guild_id`\n\n#### **3. Members Table (PostgreSQL)**\n\nStores server membership.\n\n\n| ##### Field | ##### Type | ##### Description |\n| --- | --- | --- |\n| `guild_id` | BIGINT (PK) | Server ID |\n| `user_id` | BIGINT (PK) | User ID |\n| `nickname` | VARCHAR(32) | Server-specific nickname |\n| `joined_at` | TIMESTAMP | Join time |\n| `roles` | BIGINT[] | Array of role IDs |\n\n\n**Composite Primary Key:** `(guild_id, user_id)`\n\n#### **4. Messages Table (Cassandra)**\n\nStores all messages with efficient time-range queries.\n\n\n| ##### Field | ##### Type | ##### Description |\n| --- | --- | --- |\n| `channel_id` | BIGINT (PK) | Channel ID |\n| `bucket` | TEXT (PK) | Time bucket (e.g., \"2024-01\") |\n| `message_id` | BIGINT (CK) | Snowflake message ID |\n| `author_id` | BIGINT | Author's user ID |\n| `content` | TEXT | Message text |\n| `edited_at` | TIMESTAMP | Last edit time (NULL if never) |\n| `attachments` | LIST | Attachment URLs |\n\n\n**Partition Key:** `(channel_id, bucket)` **Clustering Key:** `message_id DESC`\n\n**Why bucketing?**\n\n- Prevents unbounded partition growth\n- Channels can have millions of messages over years\n- Each bucket contains ~1 month of messages\n- Query: Get messages for January 2024 in channel X\n\n#### **5. Users Table (PostgreSQL)**\n\nStores user account information.\n\n\n| ##### Field | ##### Type | ##### Description |\n| --- | --- | --- |\n| `user_id` | BIGINT (PK) | Snowflake ID |\n| `username` | VARCHAR(32) | Username |\n| `discriminator` | CHAR(4) | #0000 tag |\n| `email` | VARCHAR(255) | Email address |\n| `avatar_url` | TEXT | Avatar image URL |\n| `created_at` | TIMESTAMP | Account creation |\n\n\n#### **6. Relationships Table (PostgreSQL)**\n\nStores friend relationships and blocks.\n\n\n| ##### Field | ##### Type | ##### Description |\n| --- | --- | --- |\n| `user_id` | BIGINT (PK) | User ID |\n| `target_id` | BIGINT (PK) | Related user ID |\n| `type` | SMALLINT | 1=friend, 2=blocked, 3=pending |\n| `created_at` | TIMESTAMP | Relationship creation |\n\n\n---\n\n# 6. Design Deep Dive\n\nNow that we have the high-level architecture and database schema in place, let's dive deeper into some critical design choices.\n\n## 6.1 Scaling WebSocket Connections\n\nManaging 10 million concurrent WebSocket connections is one of the biggest challenges. Let's explore how to scale this.\n\n### Connection Capacity per Server\n\nA single server can handle a limited number of connections:\n\n\n| ##### Resource | ##### Limit | ##### Impact |\n| --- | --- | --- |\n| File descriptors | ~1M per process | Each connection = 1 FD |\n| Memory | ~50KB per connection | 100K connections = 5GB RAM |\n| CPU | Heartbeats, serialization | Varies by message volume |\n\n\n**Practical limit:** 50,000-100,000 connections per Gateway server.\n\nFor 10M connections: **100-200 Gateway servers** required.\n\n### Gateway Server Architecture\n\n\n```mermaid\nflowchart TB\n subgraph Gateway[\"Gateway Server\"]\n WS[WebSocket Handler]\n Auth[Auth Middleware]\n Router[Event Router]\n Compress[Compression
zlib-stream]\n HeartBeat[Heartbeat Manager]\n end\n\n subgraph Connections[\"Connection Pool\"]\n C1[Connection 1]\n C2[Connection 2]\n CN[Connection N]\n end\n\n subgraph External[\"External Services\"]\n PubSub[Pub/Sub]\n Services[Backend Services]\n Session[Session Registry]\n end\n\n C1 --> WS\n C2 --> WS\n CN --> WS\n\n WS --> Auth\n Auth --> Router\n Router --> Compress\n WS --> HeartBeat\n\n Router --> PubSub\n Router --> Services\n WS --> Session\n\n style WS fill:#00ceff,stroke:#000,color:#000\n style Router fill:#38d9a9,stroke:#000,color:#000\n style PubSub fill:#ff8787,stroke:#000,color:#000\n```\n\n\n### Load Balancing WebSocket Connections\n\n**Challenge:** WebSockets are long-lived connections. Standard round-robin doesn't work well because connections aren't evenly distributed over time.\n\n**Solutions:**\n\n1. **Least Connections:** Route new connections to server with fewest active connections.\n\n1. **Consistent Hashing:** Hash user ID to determine Gateway server. Helps with session affinity and reconnection.\n\n1. **Two-Tier:**\n\n- L4 load balancer distributes TCP connections\n- Gateway servers handle WebSocket upgrade\n\n\n```mermaid\nflowchart LR\n Client[Client]\n DNS[DNS]\n L4[L4 Load Balancer
Least Connections]\n\n subgraph Gateways\n GW1[Gateway 1
45K conns]\n GW2[Gateway 2
52K conns]\n GW3[Gateway 3
48K conns]\n end\n\n Client --> DNS\n DNS --> L4\n L4 --> GW1\n L4 --> GW2\n L4 --> GW3\n\n style L4 fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n### Session Registry\n\nTrack which users are connected to which Gateway servers:\n\n\n```shell\n# Redis Hash\nKey: sessions:{user_id}\nValue: {\n \"gateway_id\": \"gateway-us-east-1-042\",\n \"session_id\": \"abc123\",\n \"connected_at\": 1699900000000\n}\n```\n\n\n**Uses:**\n\n- Route DM notifications to correct Gateway\n- Handle reconnection to same Gateway (session resume)\n- Presence updates\n\n### Connection Resumption\n\nWhen a client temporarily disconnects (network switch, sleep):\n\n1. Client reconnects with session ID and last sequence number\n2. Gateway looks up session in registry\n3. If session still valid, replay missed events\n4. Client continues without re-identifying\n\nThis avoids expensive re-initialization and provides seamless experience.\n\n---\n\n## 6.2 Message Fanout at Scale\n\nWhen a message is sent to a channel with 100,000 online members, how do we deliver it efficiently?\n\n### The Fanout Challenge\n\n**Scenario:**\n\n- Channel has 500,000 members\n- 100,000 are online (20%)\n- These 100,000 users are spread across 100 Gateway servers\n- Each message must reach all 100,000 users\n\n**Naive approach (push to each user individually):**\n\n- 1 message → 100,000 individual pushes\n- At 11,600 messages/second, this becomes 1.16 billion pushes/second\n- Completely infeasible\n\n### Solution: Hierarchical Pub/Sub\n\nUse channel-based subscriptions, not user-based:\n\n\n```mermaid\nflowchart TB\n Sender[Sender]\n MsgService[Message Service]\n\n subgraph PubSub[\"Pub/Sub Layer\"]\n Topic[Channel Topic
channel:123456]\n end\n\n subgraph Gateways[\"Subscribed Gateways\"]\n GW1[Gateway 1
5K users in channel]\n GW2[Gateway 2
3K users in channel]\n GW3[Gateway 3
2K users in channel]\n end\n\n subgraph Users[\"Users\"]\n U1[1000s of users]\n U2[1000s of users]\n U3[1000s of users]\n end\n\n Sender --> MsgService\n MsgService --> Topic\n Topic --> GW1\n Topic --> GW2\n Topic --> GW3\n GW1 --> U1\n GW2 --> U2\n GW3 --> U3\n\n style Topic fill:#ff8787,stroke:#000,color:#000\n style MsgService fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n**How it works:**\n\n1. Message Service publishes to `channel:{channel_id}` topic\n2. Only Gateway servers with users in that channel subscribe to the topic\n3. Each Gateway receives **one copy** of the message\n4. Gateway locally fans out to its connected users in that channel\n\n**Fanout reduction:**\n\n- Instead of 100,000 pushes from Pub/Sub\n- Only ~100 pushes to Gateway servers\n- Each Gateway handles local fanout (in-memory, fast)\n\n### Gateway Channel Membership\n\nEach Gateway maintains local state of which connected users are in which channels:\n\n\n```python\n# Gateway internal state\nchannel_members = {\n \"channel_123\": {\"user_1\", \"user_2\", \"user_3\", ...}, # Set of user IDs\n \"channel_456\": {\"user_1\", \"user_4\", ...},\n}\n\nuser_connections = {\n \"user_1\": WebSocketConnection,\n \"user_2\": WebSocketConnection,\n ...\n}\n```\n\n\nWhen message arrives for `channel_123`:\n\n\n```python\ndef handle_message(channel_id, message):\n user_ids = channel_members.get(channel_id, set())\n for user_id in user_ids:\n conn = user_connections.get(user_id)\n if conn:\n conn.send(message)\n```\n\n\n### Pub/Sub Technology Choice\n\n\n| ##### Option | ##### Throughput | ##### Latency | ##### Durability | ##### Best For |\n| --- | --- | --- | --- | --- |\n| **Redis Pub/Sub** | Very High | < 1ms | None | Real-time, ephemeral |\n| **Redis Streams** | High | < 5ms | Optional | Real-time with replay |\n| **Kafka** | Very High | 5-50ms | Strong | Audit, replay |\n| **NATS** | Very High | < 1ms | Optional | Cloud-native |\n\n\n**Discord uses Redis Pub/Sub** for real-time messaging because:\n\n- Lowest latency (critical for chat)\n- High throughput\n- Messages are already persisted separately\n- No need for Pub/Sub durability\n\n---\n\n## 6.3 Message Storage with Cassandra\n\nStoring 1 billion messages per day requires a database optimized for time-series writes.\n\n### Why Cassandra?\n\n\n| ##### Requirement | ##### Cassandra Capability |\n| --- | --- |\n| High write throughput | Distributed writes, no single leader |\n| Time-range queries | Clustering keys with time ordering |\n| Horizontal scaling | Linear scale with more nodes |\n| High availability | No single point of failure |\n\n\n### Partition Strategy\n\n**Problem:** A single channel could have millions of messages over years. One massive partition is inefficient.\n\n**Solution:** Bucket messages by time period.\n\n\n```shell\nPartition Key: (channel_id, bucket)\nClustering Key: message_id DESC\n\nExample partitions:\n- (channel_123, \"2024-01\") → January 2024 messages\n- (channel_123, \"2024-02\") → February 2024 messages\n```\n\n\n**Bucket size considerations:**\n\n- Too small (daily): Many partitions, complex queries across days\n- Too large (yearly): Partitions grow too big\n- Sweet spot: Monthly buckets work well\n\n### Query Patterns\n\n**Get recent messages:**\n\n\n```sql\nSELECT * FROM messages\nWHERE channel_id = 123456\n AND bucket = '2024-01'\nORDER BY message_id DESC\nLIMIT 50;\n```\n\n\n**Load more (pagination):**\n\n\n```sql\nSELECT * FROM messages\nWHERE channel_id = 123456\n AND bucket = '2024-01'\n AND message_id < 1234567890123456789\nORDER BY message_id DESC\nLIMIT 50;\n```\n\n\n**Cross-bucket query (spanning months):**\n\n\n```python\ndef get_messages(channel_id, before_id, limit=50):\n messages = []\n bucket = get_bucket_for_message_id(before_id)\n\n while len(messages) < limit:\n result = query_bucket(channel_id, bucket, before_id, limit - len(messages))\n messages.extend(result)\n\n if len(result) < requested:\n bucket = get_previous_bucket(bucket)\n if bucket is None:\n break # No more history\n before_id = None # Start from end of previous bucket\n\n return messages\n```\n\n\n### Caching Recent Messages\n\nMost message reads are for recent messages in active channels:\n\n\n```shell\n# Redis List per channel\nKey: messages:{channel_id}\nValue: Last 100 messages (serialized)\nTTL: 1 hour (refreshed on new message)\n```\n\n\n**Cache hit rate:** 90%+ for active channels\n\n- New messages added to cache immediately\n- Scrolling back hits cache for recent history\n- Only deep history requires Cassandra query\n\n---\n\n## 6.4 Voice Server Architecture (SFU)\n\nThe Selective Forwarding Unit (SFU) is the core of voice infrastructure. Let's dive deeper into how it works.\n\n### SFU Operation\n\n\n```mermaid\nflowchart TB\n subgraph Participants\n A[User A
Speaking]\n B[User B
Listening]\n C[User C
Speaking]\n D[User D
Listening]\n end\n\n subgraph SFU[\"Voice Server (SFU)\"]\n Recv[Receive
Streams]\n Route[Routing
Logic]\n Send[Send
Streams]\n end\n\n A -->|Audio| Recv\n C -->|Audio| Recv\n Recv --> Route\n Route --> Send\n Send -->|A+C Audio| B\n Send -->|A+C Audio| D\n Send -->|C Audio| A\n Send -->|A Audio| C\n\n style Recv fill:#00ceff,stroke:#000,color:#000\n style Route fill:#9775fa,stroke:#000,color:#000\n style Send fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n**How SFU routing works:**\n\n1. Each user sends ONE upstream audio stream\n2. SFU receives all incoming streams\n3. SFU selectively forwards streams to each user:\n\n- User A receives audio from B, C, D (not their own)\n- User B receives audio from A, C, D\n\n1. No mixing on server, clients mix locally\n\n### Bandwidth Optimization\n\n**Per-user bandwidth:**\n\n- Upstream: 64 kbps (Opus codec, one stream)\n- Downstream: 64 kbps × (N-1) participants\n\n**For 10-person channel:**\n\n- Each user: 64 kbps up + 576 kbps down = ~640 kbps total\n\n**For 100-person channel:**\n\n- Naive: 64 kbps up + 6.3 Mbps down = impractical!\n\n**Solution: Simulcast + Selective Subscription**\n\n1. **Simulcast:** Speakers send multiple quality levels (high/medium/low)\n2. **Voice Activity Detection:** Only forward audio from active speakers\n3. **Priority:** Forward high quality for recent speakers, low for others\n4. **Limit:** Only forward top N active speakers (e.g., 25)\n\n\n```mermaid\nflowchart LR\n subgraph Speakers[\"100 Users\"]\n Active[3 Active
Speakers]\n Inactive[97 Inactive
Listeners]\n end\n\n subgraph SFU[\"SFU Logic\"]\n VAD[Voice Activity
Detection]\n Select[Select Top
Speakers]\n end\n\n subgraph Output[\"Output Streams\"]\n Out[3 Audio Streams
192 kbps total]\n end\n\n Active --> VAD\n Inactive -.->|Silence| VAD\n VAD --> Select\n Select --> Out\n\n style VAD fill:#ffa94d,stroke:#000,color:#000\n style Select fill:#9775fa,stroke:#000,color:#000\n```\n\n\n### Voice Server Capacity\n\nSingle voice server capacity:\n\n- ~500-1000 concurrent voice participants\n- Limited by CPU (packet routing) and bandwidth\n\n**For large voice channels (500+ users):**\n\n- Use cascaded SFUs\n- Primary SFU handles speakers\n- Secondary SFUs relay to listeners in different regions\n\n### Audio Codec: Opus\n\nDiscord uses Opus codec for voice:\n\n\n| ##### Property | ##### Value |\n| --- | --- |\n| Bitrate | 6-510 kbps (adaptive) |\n| Frame size | 20ms typical |\n| Latency | < 30ms codec latency |\n| Features | Voice activity detection, error correction |\n\n\n**Adaptive bitrate:** Quality adjusts based on network conditions\n\n- Good network: 64 kbps, high quality\n- Poor network: 24 kbps, acceptable quality\n\n---\n\n## 6.5 Handling Large Servers\n\nSome Discord servers have millions of members. This creates unique challenges.\n\n### Member List Challenge\n\n**Problem:** Loading 1 million members into memory is impractical.\n\n**Solution:** Lazy member loading\n\n\n```mermaid\nflowchart TB\n subgraph Client[\"Client View\"]\n Visible[Visible Members
~100 users]\n Roles[Role Sections
Online/Offline counts]\n end\n\n subgraph Server[\"Server\"]\n MemberService[Member Service]\n Cache[Member Cache
Active members]\n DB[(Member DB)]\n end\n\n Visible -->|1. Request visible| MemberService\n MemberService --> Cache\n MemberService --> DB\n Roles -->|2. Request counts| MemberService\n\n style Cache fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Strategy:**\n\n1. **Chunked loading:** Request members in batches as user scrolls\n2. **Role-based grouping:** Show role headers with counts, load members on expand\n3. **Search:** For finding specific members, use server-side search\n4. **Cache active members:** Keep recently active/online members in cache\n\n### Permission Calculation\n\nLarge servers have complex permission hierarchies:\n\n- Server-level permissions\n- Role-based permissions (multiple roles per user)\n- Channel-level permission overwrites\n- Category-level permissions\n\n**Computing permissions on every action is expensive.**\n\n**Solution:** Cached computed permissions\n\n\n```python\n# Cache computed permissions per user per channel\nKey: perms:{guild_id}:{user_id}:{channel_id}\nValue: Bitfield of permissions\nTTL: 5 minutes (invalidate on role/permission change)\n```\n\n\n**Permission computation:**\n\n\n```python\ndef compute_permissions(user_id, channel_id):\n guild_id = get_guild_for_channel(channel_id)\n\n # Start with @everyone permissions\n perms = get_role_permissions(guild_id, everyone_role_id)\n\n # Apply each role's permissions (OR together)\n for role_id in get_user_roles(guild_id, user_id):\n role_perms = get_role_permissions(guild_id, role_id)\n perms |= role_perms\n\n # Apply channel overwrites\n overwrites = get_channel_overwrites(channel_id)\n for overwrite in overwrites:\n if overwrite.applies_to(user_id):\n perms &= ~overwrite.deny\n perms |= overwrite.allow\n\n return perms\n```\n\n\n### Gateway Intents\n\nFor large servers, sending all events to all clients is wasteful.\n\n**Gateway Intents:** Clients declare which events they want to receive.\n\n\n```python\n# Client specifies intents on IDENTIFY\nintents = {\n \"GUILD_MESSAGES\": True,\n \"GUILD_MEMBERS\": False, # Don't send member updates\n \"GUILD_PRESENCES\": False, # Don't send presence updates\n \"MESSAGE_CONTENT\": True,\n}\n```\n\n\n**Benefits:**\n\n- Clients only receive relevant events\n- Reduces Gateway bandwidth significantly\n- Especially important for bot clients in large servers\n\n---\n\n## 6.6 Global Infrastructure\n\nDiscord serves users worldwide with low latency. This requires careful infrastructure design.\n\n### Regional Architecture\n\n\n```mermaid\nflowchart TB\n subgraph US[\"US Region\"]\n US_GW[Gateway Cluster]\n US_Voice[Voice Servers]\n US_Cache[Regional Cache]\n end\n\n subgraph EU[\"EU Region\"]\n EU_GW[Gateway Cluster]\n EU_Voice[Voice Servers]\n EU_Cache[Regional Cache]\n end\n\n subgraph Asia[\"Asia Region\"]\n Asia_GW[Gateway Cluster]\n Asia_Voice[Voice Servers]\n Asia_Cache[Regional Cache]\n end\n\n subgraph Global[\"Global Services\"]\n MsgDB[(Message DB
Multi-region)]\n MetaDB[(Metadata DB
Primary + Replicas)]\n GlobalPubSub[Global Pub/Sub]\n end\n\n US_GW <--> GlobalPubSub\n EU_GW <--> GlobalPubSub\n Asia_GW <--> GlobalPubSub\n\n US_GW --> US_Cache\n EU_GW --> EU_Cache\n Asia_GW --> Asia_Cache\n\n US_Cache --> MsgDB\n EU_Cache --> MsgDB\n Asia_Cache --> MsgDB\n\n style GlobalPubSub fill:#ff8787,stroke:#000,color:#000\n style MsgDB fill:#69db7c,stroke:#000,color:#000\n```\n\n\n### Routing Strategy\n\n**Gateway Connections:**\n\n- Users connect to geographically closest Gateway cluster\n- DNS-based routing (GeoDNS) or Anycast\n- Gateways communicate via Global Pub/Sub\n\n**Voice Connections:**\n\n- Voice server assigned based on participant locations\n- For global voice channels, use server in geographic center\n- Or use cascaded SFUs for lowest latency per region\n\n**Database:**\n\n- Cassandra (messages): Multi-region, local reads, global writes\n- PostgreSQL (metadata): Primary in one region, read replicas globally\n\n### Cross-Region Messaging\n\nWhen User A (US) sends message to User B (EU):\n\n1. A's message goes to US Gateway\n2. US Gateway → Message Service → Pub/Sub\n3. Global Pub/Sub replicates to EU region\n4. EU Gateway receives message\n5. EU Gateway pushes to User B\n\n**Latency:** ~100-200ms for cross-region (acceptable for text)\n\n---\n\n# References\n\n- [How Discord Stores Billions of Messages](undefined) - Discord's Cassandra architecture\n- [How Discord Scaled to 10 Million Concurrent Users](undefined) - Scaling Elixir Gateways\n- [How Discord Handles Two Million Concurrent Voice Users](undefined) - Voice infrastructure details\n- [Real-time Communication with WebRTC](undefined) - WebRTC documentation\n- [Designing Data-Intensive Applications](undefined) by Martin Kleppmann - Distributed systems fundamentals\n- [Opus Codec](undefined) - Audio codec used by Discord","pageType":"system-design-interviews"}

Design Discord

Ashish Pratap Singh

Get Premium