{"title":"Design WhatsApp","description":null,"content":"\n> **QUESTION**\n>\n> #### What is WhatsApp?\n>\n> WhatsApp is a widely used instant messaging application that enables real-time communication between users through instant message delivery. Users can send text messages, media files, and other content to individuals or groups, with messages delivered within milliseconds.\n>\n> \n> \n> \n>\n> The core idea is simple: User A sends a message, and User B receives it instantly. However, achieving this at scale with billions of users, while ensuring message delivery guarantees, handling offline users, and supporting group conversations, introduces significant distributed systems challenges.\n>\n> **Other Popular Examples:** Facebook Messenger, Telegram, Signal, WeChat\n\n\nIn this chapter, we will work through the **high-level design of a messaging system like WhatsApp**.\n\nThis problem is a favorite in system design interviews because it touches on so many fundamental concepts: **real-time communication**, **persistent connections**, **message ordering**, **delivery guarantees**, and the challenges of building a truly **global-scale system**. \n\nLet's start by understanding what exactly we are building.\n\n---\n\n# 1. Clarifying Requirements\n\nBefore diving into the design, it's important to ask thoughtful questions to uncover hidden assumptions, clarify ambiguities, and define the system's scope more precisely.\n\nHere is an example of how a discussion between the candidate and the interviewer might unfold:\n\n\n> **DISCUSSION**\n>\n> **Candidate:** \"What is the expected scale? How many users and messages per day should the system support?\"\n>\n> **Interviewer:** \"Let's design for 500 million daily active users (DAU) sending an average of 40 messages per day.\"\n>\n> **Candidate:** \"Should we support only one-on-one messaging, or also group chats?\"\n>\n> **Interviewer:** \"Both. Group chats should support up to 500 members.\"\n>\n> **Candidate:** \"What types of content should messages support? Text only, or also media like images and videos?\"\n>\n> **Interviewer:** \"Focus on text messages for the core design. You can mention media handling at a high level, but detailed media processing is out of scope.\"\n>\n> **Candidate:** \"Do we need to show online/offline status and typing indicators?\"\n>\n> **Interviewer:** \"Yes, presence indicators (online/offline/last seen) are important. Typing indicators are nice-to-have.\"\n>\n> **Candidate:** \"What about message delivery guarantees? Should users see read receipts?\"\n>\n> **Interviewer:** \"Yes. Users should see when their message is delivered and when it's read. Messages should never be lost.\"\n>\n> **Candidate:** \"Should messages be stored permanently, or can they expire?\"\n>\n> **Interviewer:** \"Messages should be stored until explicitly deleted by the user. We need to support message history sync across devices.\"\n>\n> **Candidate:** \"What about end-to-end encryption?\"\n>\n> **Interviewer:** \"You can mention it conceptually, but detailed cryptographic implementation is out of scope.\"\n\n\nAfter gathering the details, we can summarize the key system requirements.\n\n\n### Functional Requirements\n\n1. **One-on-One Chat:** Users can send and receive messages in real-time with other users.\n2. **Group Chat:** Users can create groups and send messages to multiple recipients (up to 500 members).\n3. **Message Delivery Status:** Users can see delivery receipts (sent, delivered, read).\n4. **Online Presence:** Users can see if their contacts are online, offline, or their last seen time.\n5. **Message History:** Users can access their message history and sync across multiple devices.\n6. **Push Notifications:** Offline users receive push notifications for new messages.\n\n### Non-Functional Requirements\n\n1. **Low Latency:** Messages should be delivered within milliseconds for online users. Target: p99 < 100ms for delivery when the recipient is online.\n2. **High Availability:** The system must be highly available (99.99% uptime). Users expect messaging to work 24/7.\n3. **Reliability:** Messages must never be lost. Once sent, a message should eventually be delivered, even if the recipient is offline.\n4. **Scalability:** Support 500M+ daily active users and 20B+ messages per day.\n5. **Ordering:** Messages within a conversation should appear in the correct order.\n6. **Consistency:** Eventually consistent for presence. Message delivery is at-least-once with no duplicates, and messages within a conversation are ordered.\n\n\n\n> **Out of Scope**\n>\n> To keep our discussion focused, we will set aside a few features that, while important, would expand the scope beyond a single interview:\n>\n> - **Media Messages:** Images, videos, voice notes (mentioned conceptually only).\n> - **Voice/Video Calls:** Real-time audio and video communication.\n> - **End-to-End Encryption:** Detailed cryptographic implementation.\n> - **Stories/Status Updates:** Ephemeral content sharing.\n\n\n---\n\n# 2. Back-of-the-Envelope Estimation\n\nWith our requirements clear, lets understand the scale we are dealing with. In most interviews, you are not required to do a detailed estimation.\n\n\n> **Assumptions**\n>\n> We will use these baseline numbers throughout our calculations:\n>\n> - **Daily Active Users (DAU):** 500 million\n> - **Messages per user per day:** 40\n> - **Average message size:** 100 bytes (text content + metadata)\n> - **Average group size:** 20 members\n> - **Percentage of group messages:** 30%\n\n\n#### Message Throughput\n\nLet's start with the fundamental question: how many messages flow through this system?\n\n- **Total messages per day:** 500 million users x 40 messages = **20 billion messages/day**\n\nTwenty billion. That is 20,000,000,000 messages every single day. Let's convert that to something more tangible:\n\n- **Average messages per second:** 20 billion / 86,400 seconds = **~230,000 messages/second**\n- **Peak load (3x average):** **~700,000 messages/second**\n\nThe 3x multiplier accounts for peak hours when everyone is awake and chatting. Traffic is never uniform throughout the day.\n\nThese numbers tell us something important: we are looking at hundreds of thousands of concurrent operations per second. This rules out naive approaches like having clients poll the database for new messages. We need persistent connections, efficient routing, and aggressive caching.\n\n#### Connection Load\n\nMessaging systems differ from typical web applications here. Unlike a website where users make requests and disconnect, a messaging app needs to push messages to users the instant they arrive. \n\nThat means maintaining persistent connections with every online user.\n\n- **Concurrent connections:** If 10% of DAU are online at any time = **50 million concurrent connections**\n- **Peak concurrent connections:** **~100 million**\n\nEach of these 50 million connections requires maintaining a persistent WebSocket. This is a fundamentally different challenge from handling 50 million HTTP requests per day. These connections stay open, consuming memory and file descriptors on our servers.\n\nIf a single well-tuned server can handle 50,000 concurrent WebSocket connections (a reasonable estimate for modern hardware with proper kernel tuning), we need:\n\n\n```shell\n50 million connections / 50,000 per server = 1,000 chat servers\n```\n\n\nThat covers the average load. To absorb the ~100 million peak, we provision roughly double, around 2,000 servers. For connection handling alone, we are looking at a fleet of a couple thousand servers.\n\n#### Storage (Per Day)\n\nStorage requirements for a text-only system are more modest than you might expect:\n\n- **Message storage:** 20 billion messages x 100 bytes = **2 TB/day**\n- **Annual storage:** 2 TB x 365 = **730 TB/year** (messages alone)\n\n730 TB/year is well within reach of distributed databases like Cassandra or ScyllaDB (with replication, plan for roughly 3x). The real challenge is not capacity but access patterns: we write 230,000 messages per second while simultaneously reading history and syncing devices, so latency matters more than raw throughput.\n\n#### Bandwidth\n\nInbound is modest: 230K msg/sec x 100 bytes is about 23 MB/sec. Outbound is higher because group messages fan out to multiple devices, but even so we stay in the range of hundreds of megabytes per second, which modern network infrastructure handles easily. Bandwidth is not the bottleneck for text messaging.\n\n---\n\n# 3. Core APIs\n\nDefining the APIs early forces us to think concretely about what users can do and what data flows through the system. A messaging system's API is unusual: most real-time communication happens over persistent WebSocket connections rather than HTTP request-response, but we still need REST endpoints for operations that do not require instant delivery, like fetching message history.\n\n\n```mermaid\nflowchart LR\n subgraph \"Real-time (WebSocket)\"\n W1[Send Message]:::primary\n W2[Receive Message]:::primary\n W3[Typing Indicator]:::primary\n end\n\n subgraph \"Request/Response (REST)\"\n R1[Fetch History]:::orange\n R2[Update Status]:::orange\n R3[Get Presence]:::orange\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nLet's walk through the essential APIs.\n\n### 1. Send Message\n\n#### Endpoint: `WebSocket message or POST /messages`\n\nThis is the most frequent operation in the system. When a user taps send, this API handles getting the message from their device to ours. \n\nIn practice, this almost always goes over the WebSocket connection for lowest latency, but having a REST fallback is useful when WebSocket connections fail.\n\n##### Request Parameters\n\n\n| Parameter | Required | Description |\n|-----------|----------|-------------|\n| `sender_id` | Yes | ID of the user sending the message |\n| `recipient_id` | Yes | ID of the recipient user or group |\n| `recipient_type` | Yes | Whether the recipient is a `user` or `group` |\n| `content` | Yes | The actual message text |\n| `content_type` | No | Content kind: `text` (default), `image`, `video` |\n| `client_message_id` | Yes | Client-generated unique ID for deduplication |\n| `timestamp` | Yes | Client-side timestamp when the message was composed |\n\n\n##### Sample Response\n\n\n```json\n{\n \"message_id\": \"msg_123abc\",\n \"status\": \"sent\",\n \"server_timestamp\": \"2024-01-15T10:30:00.123Z\"\n}\n```\n\n\nThe `client_message_id` matters for reliability. If the network drops and the app retries a send, the server uses this ID to recognize the duplicate and store the message only once, which gives effectively-once delivery.\n\n### 2. Fetch Messages\n\n#### Endpoint: `GET /conversations/{conversation_id}/messages`\n\nWhen a user opens an old conversation or logs in on a new device, they need to see their message history. This endpoint retrieves messages for a conversation, typically the most recent ones first.\n\n##### Request Parameters\n\n\n| Parameter | Required | Description |\n|-----------|----------|-------------|\n| `conversation_id` | Yes | ID of the conversation to fetch |\n| `cursor` | No | Pagination cursor for fetching older messages |\n| `limit` | No | Number of messages to return (default: 50, max: 100) |\n\n\n##### Sample Response\n\n\n```json\n{\n \"messages\": [\n {\n \"message_id\": \"msg_123\",\n \"sender_id\": \"user_456\",\n \"content\": \"Hey, how are you?\",\n \"timestamp\": \"2024-01-15T10:30:00Z\",\n \"status\": \"read\"\n }\n ],\n \"next_cursor\": \"eyJtc2dfaWQiOiJtc2dfMTIyIn0=\",\n \"has_more\": true\n}\n```\n\n\nWe use cursor-based pagination rather than offset-based. With billions of messages, a query like `OFFSET 1000000` would be slow, requiring the database to skip over a million rows. Cursor-based pagination uses an indexed value (like a message ID or timestamp) to efficiently jump to the right position.\n\n### 3. Update Message Status\n\n#### Endpoint: `POST /messages/{message_id}/status`\n\nThis is what powers those checkmarks. When a message is delivered to the recipient's device or opened by the user, we need to update its status and notify the sender.\n\n##### Request Parameters\n\n\n| Parameter | Required | Description |\n|-----------|----------|-------------|\n| `message_id` | Yes | ID of the message to update |\n| `status` | Yes | New status: `delivered` or `read` |\n| `timestamp` | Yes | When the status change occurred |\n\n\nStatus updates flow in one direction: `sent → delivered → read`. We never go backwards. The timestamp helps with edge cases where status updates arrive out of order due to network delays.\n\n### 4. Get User Presence\n\n#### Endpoint: `GET /users/{user_id}/presence`\n\nReturns whether a user is currently online and, if offline, when they were last active. This powers the \"online\" indicator and \"last seen\" text in the UI.\n\n##### Sample Response\n\n\n```json\n{\n \"user_id\": \"user_456\",\n \"status\": \"offline\",\n \"last_seen\": \"2024-01-15T09:45:00Z\"\n}\n```\n\n\nPresence is intentionally kept simple. We do not need to know exactly what a user is doing, only whether they are actively connected. Privacy controls allow users to hide their last seen time, in which case we omit that field. \n\nWith our API contract defined, we have a clear picture of what the system needs to do. Now let's design the architecture that makes these APIs work at scale.\n\n---\n\n# 4. High-Level Design\n\nWe will build this system incrementally rather than presenting a full architecture diagram up front and explaining each box afterward. \n\nWe will start with the simplest possible design that solves our first requirement, then add components only as we encounter new challenges. This mirrors how you should think through the problem in an interview.\n\nOur system must ultimately satisfy three core requirements:\n\n1. **Real-time Message Delivery:** Messages should reach online recipients within milliseconds.\n2. **Offline Message Handling:** Messages for offline users need to be stored and delivered when they come back online.\n3. **Group Message Distribution:** A single message should efficiently reach all group members.\n\nBefore designing the architecture, one property shapes the rest: messaging is fundamentally a **push-based** system.\n\nThink about how a typical web application works. Your browser requests a page, the server responds, and the connection closes. If you want new data, you request again. This request-response pattern works great for most applications, but it falls apart for messaging. \n\nYou cannot expect users to constantly refresh to check for new messages. The moment a message arrives at our servers, we need to push it to the recipient's device immediately.\n\n\n```mermaid\nflowchart TD\n\n subgraph Messaging[\"Push-based\"]\n C2[Client]:::primary\n S2[Server]:::orange\n C2 <-->|\"Persistent Connection\"| S2\n S2 -->|\"Push new message\"| C2\n end\n\n subgraph Traditional[\"Pull-based\"]\n C1[Client]:::primary\n S1[Server]:::orange\n C1 -->|\"Request\"| S1\n S1 -->|\"Response\"| C1\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nThis push-based nature is why we need persistent WebSocket connections rather than traditional HTTP. And maintaining millions of persistent connections creates a whole set of challenges that we need to address.\n\nLet's start building, one requirement at a time.\n\n---\n\n## 4.1 Requirement 1: Real-time One-on-One Messaging\n\nLet's start with the simplest possible scenario: User A sends a message to User B, and User B is currently online with the app open. \n\n**What do we need to make this work?**\n\nWith both users online, we push the message to User B the instant it arrives over a persistent connection. The question is which components make that happen.\n\n### The Components We Need\n\nLet's introduce the components one by one, understanding why each exists.\n\n#### Chat Servers\n\nChat servers handle most of the system's work. Each one maintains persistent WebSocket connections with thousands of clients simultaneously.\n\nWhen User A opens the messaging app, their phone establishes a WebSocket connection to one of the chat servers. This connection stays open for as long as the app is in use. When User A sends a message, it travels over this existing connection, no need to establish a new one.\n\nBeyond holding connections, a chat server routes messages to recipients that may live on other chat servers, sends heartbeats to detect dead connections, and handles reconnection when users switch networks.\n\nChat servers are **stateful**. Unlike typical web servers where any server can handle any request, User B's messages must go to the specific chat server where User B's connection lives. If User B is connected to Chat Server 2, sending their message to Chat Server 1 will not help.\n\n\n```mermaid\nflowchart TB\n\n\tsubgraph CS3[\"Chat Server 3\"]\n U6[\"User F\"]:::primary\n U7[\"User G\"]:::primary\n end\n\n subgraph CS2[\"Chat Server 2\"]\n U4[\"User B\"]:::orange\n U5[\"User E\"]:::primary\n end\n\n subgraph CS1[\"Chat Server 1\"]\n U1[\"User A\"]:::primary\n U2[\"User C\"]:::primary\n U3[\"User D\"]:::primary\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nThis statefulness creates a routing challenge. When User A sends a message to User B, how does Chat Server 1 know that User B is on Chat Server 2?\n\n#### Session Service\n\nThis is where the Session Service comes in. It maintains a simple but critical mapping: which user is connected to which chat server.\n\nWhen User B connects to Chat Server 2, that server registers the connection: \"User B is on Chat Server 2.\" When User A wants to send a message to User B, we query the Session Service: \"Where is User B?\" It responds: \"Chat Server 2.\"\n\nWe typically implement this using **Redis** because it offers exactly what we need: fast key-value lookups with built-in expiration for handling disconnections. The data structure is simple:\n\n\n```shell\nuser_a -> chat_server_1\nuser_b -> chat_server_2\nuser_c -> chat_server_1\n...\n```\n\n\n#### Message Service\n\nWhile routing messages in real-time is essential, we also need to persist them. Users expect to see their message history. If User B's phone dies right as a message arrives, we do not want to lose it.\n\nThe Message Service persists every message to the database before attempting delivery, generates server-side message IDs and timestamps for ordering, tracks message status (sent, delivered, read), and serves message history queries for the Fetch Messages API.\n\n### Putting It Together: The Message Flow\n\nNow let's trace what happens when User A sends \"Hey, how's it going?\" to User B. Both users are online, connected to different chat servers.\n\n\n```mermaid\nsequenceDiagram\n participant A as User A\n participant CS1 as Chat Server 1\n participant SS as Session Service\n participant MS as Message Service\n participant DB as Database\n participant CS2 as Chat Server 2\n participant B as User B\n\n A->>CS1: 1. Send message to User B\n CS1->>MS: 2. Persist message\n MS->>DB: Store message\n DB-->>MS: Confirm\n MS-->>CS1: 3. Message ID + timestamp\n CS1->>SS: 4. Where is User B?\n SS-->>CS1: 5. Chat Server 2\n CS1->>CS2: 6. Forward message\n CS2->>B: 7. Push via WebSocket\n B-->>CS2: 8. ACK (delivered)\n CS2->>MS: 9. Update status\n CS2->>CS1: 10. Delivery confirmation\n CS1->>A: 11. Message delivered ✓✓\n```\n\n\nLet's walk through the flow:\n\nWhen User A taps send, the message travels over the existing WebSocket to Chat Server 1, which first asks the Message Service to persist it. Persisting before anything else means the message survives even if a later step fails.\n\nThe Message Service writes it to the database and returns a server-generated message ID and timestamp; the server's clock, not the client's, is the source of truth for ordering. With the message stored, Chat Server 1 queries the Session Service (\"Where is User B?\") and gets back \"Chat Server 2\" in under a millisecond.\n\nChat Server 1 then forwards the message to Chat Server 2, typically over gRPC, and Chat Server 2 pushes it to User B. Direct server-to-server forwarding is simple, but maintaining a full mesh across thousands of chat servers is hard to manage.\n\nA common alternative is a publish/subscribe backplane (Redis Pub/Sub or Kafka) where each server subscribes to a channel for the users it holds, so the sender publishes to the recipient's channel instead of addressing a specific server.\n\nFinally, User B's client acknowledges receipt, the status updates to \"delivered\" along the way, and User A's UI shows the double checkmark. When both users are online, this round trip completes in under 100 milliseconds.\n\nThis raises the next question: **what happens when User B is not online?**\n\n---\n\n## 4.2 Requirement 2: Handling Offline Users\n\nThe flow we designed works when both users are online. Real-world messaging is messier than that. What happens when User B's phone is in airplane mode? What if they have not opened the app in hours? What if they are in a subway tunnel with no signal?\n\nWe cannot drop the message; that would violate our reliability requirement. Users expect that once they tap send, the message eventually arrives, even if the recipient is unreachable for hours or days. So instead of pushing a message and forgetting about it, we need to track pending deliveries and retry when users come back online.\n\n### New Components for Offline Handling\n\nLet's introduce two new pieces to our architecture.\n\n#### Offline Message Handling\n\nThe message is already persisted in the message database, so durability is covered. What we still need is a way to know which messages a user has not yet received, plus a pipeline to deliver them once the user returns.\n\nWe track delivery progress with a per-user marker: the ID of the last message the user's device has acknowledged. Any message newer than that marker is pending. When the user reconnects, we read everything after their marker straight from the message store, in order.\n\nThis is the same catch-up mechanism we use for multi-device sync later in the design, so offline delivery and device sync share one path instead of two.\n\nFor the active delivery path (attempting delivery, retrying, and waking devices through push), we use a queue like **Kafka**. One detail matters here: we do not create a topic per user, which at our scale would mean hundreds of millions of topics, something Kafka does not handle well.\n\nInstead, a small number of partitioned topics carry delivery tasks, partitioned by recipient ID so each user's tasks stay ordered. The queue handles retries and backpressure; the database remains the durable source of truth for the messages themselves.\n\n#### Push Notification Service\n\nEven though we cannot deliver the message content directly to an offline user, we can still tell them something is waiting. This is where push notifications come in.\n\nThe Push Notification Service integrates with Apple Push Notification Service (APNs) for iOS and Firebase Cloud Messaging (FCM) for Android. It sends a notification like \"New message from User A\" to wake up the device, while respecting user preferences such as muted conversations and quiet hours.\n\n### The Offline Message Flow\n\nLet's trace what happens when User A sends a message but User B is offline.\n\n#### When the message is sent\n\nThe diagram traces the two paths Chat Server 1 can take after persisting the message: forward it directly if User B is online, or route it through the message queue and push notification pipeline if not.\n\n\n```mermaid\nflowchart TD\n A[User A sends message]:::primary --> CS1[Chat Server 1]\n CS1 --> MS[Message Service]\n MS --> DB[(Database)]\n\n CS1 --> SS{Session Service:
Is User B online?}:::yellow\n\n SS -->|Online| ROUTE[Route to Chat Server]\n SS -->|Offline| MQ[Message Queue]\n\n MQ --> PNS[Push Notification
Service]\n PNS --> APNS[APNs / FCM]\n APNS --> Device[User B's Device]\n\n style A fill:#00ceff,stroke:#000,color:#000\n style CS1 fill:#38d9a9,stroke:#000,color:#000\n style MS fill:#38d9a9,stroke:#000,color:#000\n style DB fill:#38d9a9,stroke:#000,color:#000\n style SS fill:#ffa94d,stroke:#000,color:#000\n style MQ fill:#69db7c,stroke:#000,color:#000\n style PNS fill:#da77f2,stroke:#000,color:#000\n style APNS fill:#f783ac,stroke:#000,color:#000\n style Device fill:#f783ac,stroke:#000,color:#000\n style ROUTE fill:#38d9a9,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n```\n\n\n1. User A sends a message to User B. The message arrives at Chat Server 1.\n2. Chat Server 1 persists the message via the Message Service. It is now safely stored.\n3. Chat Server 1 queries the Session Service: \"Where is User B?\"\n4. The Session Service finds no entry for User B, meaning they are offline.\n5. Instead of failing, Chat Server 1 leaves the message marked undelivered in the store and enqueues a delivery task for User B.\n6. The Push Notification Service sends a notification to User B's phone: \"New message from User A.\"\n7. User A sees a single checkmark (sent) but not a double checkmark (delivered) yet.\n\n#### When User B comes back online\n\nOnce User B reconnects, the chat server reads their last-delivered marker and fetches every pending message from the store in one pass, ensuring nothing sent during the offline period is skipped.\n\n\n```mermaid\nflowchart TB\n subgraph Later[\"When User B Comes Online\"]\n B_CONNECT[User B connects]:::green --> CS3[Chat Server]\n CS3 --> FETCH[Fetch messages
after last marker]\n FETCH --> MQ2[(Message Store)]\n MQ2 --> DELIVER[Deliver messages]\n end\n\n style B_CONNECT fill:#69db7c,stroke:#000,color:#000\n style CS3 fill:#38d9a9,stroke:#000,color:#000\n style FETCH fill:#38d9a9,stroke:#000,color:#000\n style DELIVER fill:#38d9a9,stroke:#000,color:#000\n style MQ2 fill:#69db7c,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n1. User B opens the app and establishes a WebSocket connection to a chat server (maybe Chat Server 3 this time).\n2. During the connection handshake, Chat Server 3 reads User B's last-delivered marker and fetches every message after it from the message store.\n3. The store returns all pending messages, ordered by time.\n4. Chat Server 3 pushes these messages to User B over the WebSocket connection.\n5. User B's client acknowledges receipt.\n6. User B's marker advances to the latest delivered message, and the message statuses are updated to \"delivered.\"\n7. Back at User A's device, the checkmarks update from single to double.\n\nWith this design, the message is never lost. Whether User B comes online in 10 seconds or 10 days, the message will be waiting, because it is persisted to the database before anything else and the queue is only an optimization for fast delivery.\n\n---\n\n## 4.3 Requirement 3: Group Messaging\n\nSo far we have handled one-on-one messaging. But groups introduce a new challenge that fundamentally changes our design: **fanout**.\n\nConsider this scenario: User A sends \"Happy New Year!\" to a group with 500 members, our maximum size. That single message needs to reach 500 different devices, potentially scattered across dozens of different chat servers, some members online and some offline, some on fast WiFi and some on spotty mobile networks.\n\nWith one-on-one messaging, one input means one output. With groups, one input means many outputs. This multiplier effect is called fanout, and it can easily overwhelm a naive implementation.\n\n### Understanding the Fanout Problem\n\nLet's visualize what happens when a message goes to a group:\n\n\n```mermaid\nflowchart TD\n MSG[\"User A sends:
'Hello everyone!'\"]:::primary\n\n MSG --> FANOUT{Fanout Required}:::yellow\n\n FANOUT --> M1[Member 1
Chat Server 1]:::secondary\n FANOUT --> M2[Member 2
Chat Server 1]:::secondary\n FANOUT --> M3[Member 3
Chat Server 2]:::secondary\n FANOUT --> M4[Member 4
Chat Server 2]:::secondary\n FANOUT --> M5[Member 5
Chat Server 3]:::secondary\n FANOUT --> M6[\"... Member 500
Chat Server N\"]:::secondary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef yellow fill:#ffd43b,stroke:#000,color:#000\n```\n\n\nIf User A sends a message to a group with 500 members (our maximum), and the sender's chat server has to individually deliver to all 500, we have a problem. That server becomes a bottleneck, and if it crashes mid-delivery, some members get the message and some do not. Large groups would also create noticeable delays.\n\nThere are several ways to handle fanout. Let's examine each and understand their trade-offs.\n\n### Approach 1: Sender-Side Fanout\n\nThe simplest approach is to have the sender's chat server do all the work. When User A sends a group message, Chat Server 1 looks up all group members, finds their chat servers, and delivers to each one.\n\n\n```mermaid\nflowchart TD\n A[User A]:::primary --> CS1[Chat Server 1]:::secondary\n CS1 --> M1[Member 1]:::secondary\n CS1 --> M2[Member 2]:::secondary\n CS1 --> M3[Member 3]:::secondary\n CS1 --> M4[...]:::secondary\n CS1 --> M500[Member 500]:::secondary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n#### How it works\n\n1. Chat Server 1 queries the Group Service for all members of the group\n2. For each member, it queries the Session Service to find their chat server\n3. It forwards the message to each destination chat server\n4. Each chat server delivers to their connected members\n\n\n### Pros\n\n- Simple to implement and reason about\n- Low latency for small groups since there is no intermediary\n- No additional infrastructure required\n\n### Cons\n\n- A single server becomes the bottleneck for large groups\n- If that server crashes mid-fanout, delivery is incomplete\n- A 500-member group could take several seconds to process\n\n\nThis approach works fine for small groups (under 50-100 members), which are the majority of groups in typical usage patterns.\n\n### Approach 2: Message Queue Fanout\n\nFor larger groups, we can use a message queue to distribute the work across multiple workers.\n\n\n```mermaid\nflowchart LR\n A[User A]:::primary --> CS1[Chat Server 1]:::secondary\n CS1 --> K[Kafka Topic:
group_123]:::teal\n\n K --> W1[Worker 1]:::orange\n K --> W2[Worker 2]:::orange\n K --> W3[Worker 3]:::orange\n\n W1 --> M1[Members 1-167]:::secondary\n W2 --> M2[Members 168-334]:::secondary\n W3 --> M3[Members 335-500]:::secondary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n#### How it works\n\n1. Chat Server 1 publishes the message to a Kafka topic for the group\n2. Multiple worker processes consume from this topic in parallel\n3. Each worker is responsible for delivering to a subset of group members\n4. The work is automatically distributed based on worker capacity\n\n\n### Pros\n\n- Scales horizontally by adding more workers\n- Resilient to individual worker failures (Kafka retries automatically)\n- No single point of bottleneck\n\n### Cons\n\n- Adds latency since messages pass through the queue\n- More complex infrastructure to manage\n- Overkill for small groups where direct delivery is faster\n\n\n### Approach 3: Hybrid Approach (Recommended)\n\nWe can combine both approaches, choosing based on group size:\n\n\n```mermaid\nflowchart TB\n MSG[Group Message]:::primary\n\n MSG --> CHECK{Group Size?}:::orange\n\n CHECK -->|\"< 100 members\"| DIRECT[\"Direct Fanout
from Sender's Server\"]:::green\n CHECK -->|\">= 100 members\"| QUEUE[\"Kafka Queue
Distributed Fanout\"]:::teal\n\n DIRECT --> D1[Member 1]:::secondary\n DIRECT --> D2[Member 2]:::secondary\n DIRECT --> D3[...]:::secondary\n\n QUEUE --> W1[\"Worker 1
Members 1-250\"]:::secondary\n QUEUE --> W2[\"Worker 2
Members 251-500\"]:::secondary\n QUEUE --> W3[\"Worker N
Members...\"]:::secondary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\n- **Small groups (under 100 members):** Use direct fanout from the sender's server. This is faster and simpler, and most groups fall into this category.\n- **Large groups (100+ members):** Route through Kafka for distributed fanout. The added latency is acceptable for groups where reliability and scalability matter more.\n\nThe threshold of 100 is not fixed; it is a tunable parameter based on your server capacity. Different group sizes warrant different delivery strategies: low latency for the common case of small groups, and scalable distributed fanout for large ones.\n\n### The Complete Group Message Flow\n\nLet's put it all together and trace a group message from send to delivery:\n\n\n```mermaid\nflowchart TD\n A[User A sends
group message]:::primary --> CS1[Chat Server 1]\n CS1 --> MS[Message Service]\n CS1 --> GS[Group Service:
Get members]\n GS --> CS1\n CS1 --> SS[Session Service:
Get member locations]\n SS --> CS1\n\n CS1 --> CS2[Chat Server 2]\n CS1 --> CS3[Chat Server 3]\n CS1 --> MQ[Message Queue
delivery pipeline]\n\n CS2 --> B[User B]:::rose\n CS2 --> C[User C]:::rose\n CS3 --> D[User D]:::rose\n\n style A fill:#00ceff,stroke:#000,color:#000\n style CS1 fill:#38d9a9,stroke:#000,color:#000\n style MS fill:#38d9a9,stroke:#000,color:#000\n style GS fill:#38d9a9,stroke:#000,color:#000\n style SS fill:#ffa94d,stroke:#000,color:#000\n style CS2 fill:#38d9a9,stroke:#000,color:#000\n style CS3 fill:#38d9a9,stroke:#000,color:#000\n style MQ fill:#69db7c,stroke:#000,color:#000\n style B fill:#f783ac,stroke:#000,color:#000\n style C fill:#f783ac,stroke:#000,color:#000\n style D fill:#f783ac,stroke:#000,color:#000\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef rose fill:#f783ac,stroke:#000,color:#000\n```\n\n\n#### Step by step\n\n1. **Persist first:** User A sends a message to the group. Chat Server 1 immediately persists it via the Message Service with the `group_id` instead of a single recipient. The message is now durable.\n2. **Get members:** Chat Server 1 queries the Group Service: \"Who are the members of this group?\" The Group Service returns the list of member IDs.\n3. **Find member locations:** For each member, we need to know where they are connected (or if they are offline). Instead of making 50 individual queries, we batch this: \"Where are users [B, C, D, E, ...]?\" The Session Service returns a map of user IDs to chat server locations.\n4. **Batch by server:** Instead of forwarding to each member individually, we group members by their chat server. If members B, C, and E are all on Chat Server 2, we send one message to Chat Server 2 with all three recipient IDs. This reduces network overhead.\n5. **Parallel delivery:** Chat Server 1 sends messages to Chat Server 2 and Chat Server 3 in parallel. Each receiving chat server pushes the message to its connected members.\n6. **Handle offline members:** For members who are offline, the message goes to their queue. When they reconnect, they will receive it along with any other pending messages.\n\nThis flow handles groups of any size efficiently. For small groups, it completes in tens of milliseconds. For larger groups using the queue-based approach, delivery might take a bit longer but remains reliable.\n\n---\n\n## 4.4 Putting It All Together\n\nWe have now addressed each requirement incrementally. Let's step back and see the complete picture. This is the architecture you would draw on the whiteboard after explaining each component:\n\n\n```mermaid\nflowchart TB\n subgraph Clients[\" \"]\n direction LR\n WEB[Web App]:::client\n MOB[Mobile App]:::client\n end\n\n subgraph Edge[\"Edge Layer\"]\n direction LR\n LB[Load Balancer]:::edge\n APIGW[API Gateway]:::edge\n end\n\n CHAT[\"Chat Servers
(1 ... N)\"]:::chat\n\n subgraph Services[\"Services\"]\n direction LR\n MS[Message
Service]:::svc\n GS[Group
Service]:::svc\n US[User
Service]:::svc\n PNS[Push Notification
Service]:::svc\n end\n\n subgraph Data[\"Data Layer\"]\n direction LR\n REDIS[(Redis
Sessions + Presence)]:::cache\n KAFKA[[Kafka
Delivery Queue]]:::queue\n CASS[(Cassandra
Messages)]:::db\n PG[(PostgreSQL
Users + Groups)]:::db\n end\n\n WEB -->|WebSocket| LB\n MOB -->|WebSocket| LB\n WEB -->|REST| APIGW\n MOB -->|REST| APIGW\n\n LB --> CHAT\n APIGW --> MS\n APIGW --> GS\n APIGW --> US\n\n CHAT --> MS\n CHAT --> REDIS\n CHAT --> KAFKA\n KAFKA --> PNS\n\n MS --> CASS\n GS --> PG\n US --> PG\n\n classDef client fill:#00ceff,stroke:#000,color:#000\n classDef edge fill:#38d9a9,stroke:#000,color:#000\n classDef chat fill:#ffa94d,stroke:#000,color:#000\n classDef svc fill:#69db7c,stroke:#000,color:#000\n classDef cache fill:#ff8787,stroke:#000,color:#000\n classDef queue fill:#ffd43b,stroke:#000,color:#000\n classDef db fill:#3bc9db,stroke:#000,color:#000\n```\n\n\nThe system breaks into clear layers. Clients (mobile and web) are WebSocket clients. The edge layer load-balances incoming connections, using sticky sessions (or consistent hashing by user ID) so a connection stays on the same chat server.\n\nThe chat layer is stateful, since each server holds specific users' connections, while the service layer (messages, groups, users, presence) is stateless and scales horizontally. The data layer combines Redis (sessions and presence), Kafka (delivery pipeline), Cassandra (message history), and PostgreSQL (user and group data).\n\nWith the high-level architecture clear, let's look at how we store all this data efficiently.\n\n---\n\n# 5. Database Design\n\nWith 20 billion messages per day and 500 million users, database choices matter. The wrong one becomes a bottleneck that is hard to fix later.\n\n## 5.1 SQL vs NoSQL\n\nA messaging system has two very different kinds of data, and each deserves its own storage strategy.\n\n### Message Data: Write-Heavy and Time-Ordered\n\nMessages have a clear access pattern:\n\n- **Write-heavy workload:** We are writing 230,000 messages per second. Every single message needs to be persisted.\n- **Simple queries:** \"Get me the last 50 messages for this conversation.\" No complex joins, no aggregations.\n- **Time-series nature:** Recent messages are accessed constantly. Messages from last year are rarely touched.\n- **No transactions needed:** A message either exists or it does not. We do not need to atomically update multiple messages.\n- **High availability is critical:** If the message database goes down, messaging stops.\n\nA **wide-column NoSQL database** like **Apache Cassandra** or **ScyllaDB** fits these patterns: log-structured storage absorbs high write throughput, adding nodes scales capacity linearly, clustering keys keep each conversation sorted by time on disk, and consistency is tunable per query.\n\n### User and Group Data: Relational and Consistent\n\nUser and group data is different:\n\n- **Complex relationships:** Which groups does this user belong to? Who are the admins of this group?\n- **Transactions:** When adding a user to a group, we need to update the group membership and the user's group list atomically\n- **Strong consistency:** If I just added you to a group, you should see it immediately\n- **Read-heavy:** User profiles are read often but updated rarely\n\nA **relational database** like **PostgreSQL** fits these needs: joins and complex filters for the relationships, ACID transactions for atomic multi-table updates, and strong consistency so reads always see the latest writes.\n\n## 5.2 Database Schema\n\nNow the schemas. Each table is shaped around its access pattern and stored in the technology best suited to it:\n\n### 1. Messages Table (Cassandra)\n\nThis is the central table in our storage layer. The schema design is driven by a single question: \"What is the most common query we need to answer?\"\n\nFor a messaging app, that query is: **\"Get the last 50 messages for this conversation, ordered by time.\"**\n\nWe design the entire table around this access pattern:\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `conversation_id` | UUID (Partition Key) | Unique identifier for the conversation |\n| `message_id` | TimeUUID (Clustering Key) | Time-based UUID for ordering |\n| `sender_id` | UUID | ID of the message sender |\n| `content` | Text | Message content |\n| `content_type` | Text | Content kind: `text`, `image`, `video` |\n| `status` | Text | Delivery status: `sent`, `delivered`, `read` |\n| `created_at` | Timestamp | Server timestamp |\n\n\n**Partition Key (**`conversation_id`**):** This determines which nodes store the data. All messages in a single conversation live together on the same nodes. When we query \"last 50 messages for conversation X\", Cassandra knows exactly which nodes to ask. This is what makes reads fast.\n\nA natural question is where `conversation_id` comes from, since the Send Message API only carries `sender_id` and `recipient_id`. We derive it deterministically so both participants resolve to the same partition. For a one-on-one chat, we sort the two user IDs and hash the pair, for example `conversation_id = hash(min(A, B) + max(A, B))`.\n\nSorting first guarantees that A messaging B and B messaging A land in the same conversation. For a group chat, the `conversation_id` is the `group_id`, which already uniquely identifies the conversation.\n\nThis is why the Send API takes a `recipient_type`: it tells the server whether to derive the ID from a user pair or use the group ID directly. The client never has to know the `conversation_id`; the server computes it on write, and the Fetch Messages API uses the same derivation to read.\n\n**Clustering Key (**`message_id`** as TimeUUID):** Within a partition (a conversation), messages are physically sorted on disk by the clustering key. A TimeUUID is a special UUID that encodes the timestamp, so messages are automatically ordered by time. Fetching \"the last 50 messages\" becomes a simple range scan, not a full table scan.\n\nTogether, these keys let the common query hit a single partition and read data that is already sorted.\n\nOne caveat: partitioning purely by `conversation_id` lets a busy, long-lived conversation grow without bound, and very large partitions hurt Cassandra's read performance. In practice we add a time bucket to the partition key, for example `(conversation_id, year_month)`, so each partition covers a bounded window. Reads for recent messages hit the current bucket, and older history pages into earlier buckets.\n\n### 2. User Conversations Table (Cassandra)\n\nWhen a user opens the app, the first thing they see is their conversation list. We need to answer: \"What are this user's recent conversations, and what was the last message in each?\"\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `user_id` | UUID (Partition Key) | User ID |\n| `last_message_at` | Timestamp (Clustering Key, DESC) | Time of last message |\n| `conversation_id` | UUID (Clustering Key) | Conversation ID |\n| `unread_count` | Integer | Number of unread messages |\n| `last_message_preview` | Text | Preview of last message |\n\n\nWe denormalize `last_message_preview` and `unread_count` directly into this table so rendering the conversation list takes one read instead of a join or a second query against the messages table. Storing data in the shape you read it, even at the cost of duplication, is a common Cassandra pattern.\n\nThe clustering key is `last_message_at` in descending order, not `conversation_id`. This matters because the conversation list is shown most-recent-first, and clustering by `last_message_at` means Cassandra returns the rows already in that order. The trade-off is that `last_message_at` changes every time a new message arrives, and a clustering key cannot be updated in place.\n\nSo on each new message we delete the old row and insert a new one at the updated position, which moves the conversation to the top of the list. Because we are rewriting the row anyway, the new row carries the recomputed `unread_count` and `last_message_preview`, so a regular column write is enough; we do not need a separate Cassandra counter.\n\n#### The cost of this table is write amplification\n\nUpdating it is a per-recipient operation, not a per-message one. A one-on-one message touches two rows (one per participant). A 500-member group message touches up to 500 rows, one for each member's conversation list.\n\nSo a single group send becomes a fanout of writes here, on top of the one durable write to the messages table.\n\nThis is an acceptable trade because the conversation list is read far more often than it is written, and these writes are small and can be applied asynchronously after the message is durably stored.\n\nFor very large or very active groups, this fanout is also a reason to cap group size and to update each member's list lazily (for example, only refreshing `unread_count` when that member is online or opens the app) rather than writing to all 500 rows on every message.\n\n### 3. Groups Table (PostgreSQL)\n\nGroup metadata lives in PostgreSQL where we can use proper relational modeling:\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `group_id` | UUID (PK) | Unique group identifier |\n| `name` | VARCHAR(100) | Group name |\n| `creator_id` | UUID (FK) | User who created the group |\n| `created_at` | Timestamp | Creation time |\n| `member_count` | Integer | Number of members |\n\n\nThe `member_count` is denormalized here even though we could compute it from the members table. This avoids a COUNT query every time we need to display group info.\n\n### 4. Group Members Table (PostgreSQL)\n\nThis is the join table that maps users to groups:\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `group_id` | UUID (PK, FK) | Group ID |\n| `user_id` | UUID (PK, FK) | User ID |\n| `role` | VARCHAR(20) | Role: `admin`, `member` |\n| `joined_at` | Timestamp | When user joined |\n\n\nThe composite primary key `(group_id, user_id)` serves two purposes:\n\n1. It ensures a user can only be in a group once (no duplicates)\n2. It creates an index that allows efficient queries in both directions: \"all members of group X\" and \"all groups user Y belongs to\"\n\nWith these tables, we can handle all group operations with standard SQL queries and proper transaction support. When a user joins a group, we update both the membership table and the group's member_count in a single transaction.\n\n### 5. Message Receipts Table (Cassandra)\n\nThe single `status` field on the messages table is enough for a one-on-one chat, where a message has exactly one recipient. Group chats break this assumption. A message sent to 50 people is delivered and read by each member at a different time, and we cannot represent \"delivered to 30 of 50, read by 12\" with one column.\n\nWe track per-recipient status in a separate table:\n\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `message_id` | TimeUUID (Partition Key) | The message being tracked |\n| `user_id` | UUID (Clustering Key) | A recipient of the message |\n| `status` | Text | Per-recipient status: `delivered`, `read` |\n| `updated_at` | Timestamp | When this recipient's status last changed |\n\n\nPartitioning by `message_id` keeps all receipts for a single message together, so answering \"who has read this?\" is a single-partition read. For a one-on-one chat this table holds one row per message; for a group, one row per member.\n\nThe sender's UI shows the double checkmark once every recipient has a `delivered` row, and blue checkmarks once everyone has a `read` row. The messages table itself stays small, storing each message only once.\n\nNow let's move on to the deep dive into specific challenges.\n\n---\n\n# 6. Design Deep Dive\n\nThe high-level architecture gives us the skeleton. The deep dive is where you show that you understand how the components work and why one approach beats another. Let's explore the more challenging parts of building a messaging system.\n\n## 6.1 WebSocket vs Long Polling vs Server-Sent Events\n\nWe have used WebSocket connections throughout this design, but why WebSocket specifically? There are several ways to achieve real-time communication between client and server, each with different trade-offs in latency, resource usage, and complexity.\n\n### Approach 1: HTTP Long Polling\n\nLong polling is the oldest technique for achieving real-time-like behavior with plain HTTP. It predates WebSockets and was the backbone of early real-time web apps like Gmail's chat.\n\n**The idea is simple:** the client makes an HTTP request asking \"any new messages for me?\" Instead of responding immediately with \"no,\" the server holds the connection open. If a new message arrives while the connection is open, the server responds with it immediately.\n\nIf nothing happens for 30-60 seconds, the server responds with an empty result, and the client immediately makes another request.\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n\n C->>S: HTTP Request (any new messages?)\n Note over S: Hold connection open
(up to 30-60 seconds)\n S-->>C: Response (here's a new message!)\n\n C->>S: HTTP Request (any new messages?)\n Note over S: Hold connection...
waiting for data...\n Note over S: Timeout reached\n S-->>C: Empty response (nothing new)\n\n C->>S: HTTP Request (any new messages?)\n Note over S: Message arrives!\n S-->>C: Response (new message!)\n```\n\n\nThe client creates a continuous loop of requests, effectively creating a \"persistent\" connection using standard HTTP semantics.\n\n\n### Pros\n\n- Works everywhere, including through corporate proxies and strict firewalls that block other protocols\n- Uses standard HTTP infrastructure, so load balancers and caching work as expected\n- Simple fallback when more modern approaches are blocked\n\n### Cons\n\n- Each request cycle involves TCP connection setup, HTTP headers, and potential TLS handshakes, all overhead that adds up\n- There is inherent latency: a message might arrive just after the last response, requiring the user to wait for the next polling cycle\n- Servers hold many mostly-idle connections, wasting resources\n\n\nLong polling got us through the early web era, but it is not ideal for a modern messaging system with millions of concurrent users.\n\n### Approach 2: Server-Sent Events (SSE)\n\nSSE improves on long polling by establishing a true persistent connection, but only in one direction. The server can push events to the client continuously, but the client still needs to use regular HTTP requests to send data back.\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n\n C->>S: Open SSE connection\n Note over C,S: Persistent HTTP connection established\n\n S-->>C: Event: new message from Alice\n S-->>C: Event: Alice is typing...\n S-->>C: Event: new message from Alice\n\n Note over C: Need to send a message\n C->>S: POST /messages (separate HTTP request)\n\n S-->>C: Event: message delivered ✓\n```\n\n\nThink of SSE as a one-way pipe from server to client. The server can push events whenever it wants, but sending a message back requires a separate HTTP POST.\n\n\n### Pros\n\n- Eliminates the request/response cycle of long polling\n- Built-in reconnection handling when connections drop\n- Works well with HTTP/2, which can multiplex multiple streams\n\n### Cons\n\n- Fundamentally unidirectional, so chat requires a hybrid approach: SSE for receiving, HTTP for sending\n- This asymmetry adds complexity and slightly higher latency for outgoing messages\n- Not as universally supported as WebSockets in mobile SDKs\n\n\nSSE is a good fit for notification streams, live feeds, or stock tickers where the server broadcasts and the client mostly listens. For chat, where both sides constantly send data, we need something better.\n\n### Approach 3: WebSocket (Recommended)\n\nWebSocket is the modern solution. It provides a true bidirectional channel where both client and server can send messages at any time, over a single persistent TCP connection.\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n\n C->>S: HTTP Upgrade: WebSocket\n S-->>C: 101 Switching Protocols\n Note over C,S: Full-duplex connection established\n\n C->>S: Send message to Bob\n S-->>C: Message sent ✓\n\n S-->>C: New message from Alice\n C->>S: Mark as read\n\n S-->>C: Alice is typing...\n C->>S: I'm typing too...\n\n Note over C,S: Both sides can send
at any time\n```\n\n\nThe connection starts with a standard HTTP request that includes an \"Upgrade\" header. If the server supports WebSocket, it responds with 101 Switching Protocols, and from that point on, the connection is a full WebSocket. Both sides can send frames whenever they want, there is no request/response dance.\n\n\n### Pros\n\n- True bidirectional communication: either side can initiate a message at any time\n- Minimal overhead after connection: just the WebSocket frame header (as small as 2 bytes)\n- Lowest possible latency: messages are pushed instantly\n- Single connection for all traffic, reducing connection setup costs\n\n### Cons\n\n- Stateful nature complicates scaling: you need to track which server each user is on\n- Requires WebSocket-aware load balancers and proxies\n- Connection management requires heartbeats, reconnection logic, and careful error handling\n\n\n### Which Should You Choose?\n\n\n| Approach | Latency | Overhead | Bidirectional | Best For |\n|----------|---------|----------|---------------|----------|\n| **Long Polling** | High | High | No | Legacy systems, fallback |\n| **SSE** | Medium | Low | No | Notifications, live feeds |\n| **WebSocket** | Lowest | Lowest | Yes | Chat, gaming, collaboration |\n\n\n**For a messaging system, WebSocket is the best fit.** The bidirectional nature matches how chat works: both users send and receive constantly, the low latency keeps conversations feeling instant, and a single connection per user lets each server hold more of them. Its one real drawback, the stateful nature, we have already handled with the Session Service.\n\n\n> **One practical note**\n>\n> Always implement long polling as a fallback. Some corporate networks and older proxies still block WebSocket connections. Your client should detect this and gracefully fall back to long polling\n\n\n---\n\n## 6.2 Message Delivery Guarantees\n\nEveryone who has used WhatsApp knows the checkmarks: one gray for sent, two gray for delivered, two blue for read. These simple icons hide a lot of complexity. \n\nHow do we track message state reliably across unreliable networks, flaky mobile connections, and devices that go offline unpredictably?\n\nLet's break down what each state means and how we guarantee correct transitions:\n\n1. **Sent (single gray checkmark):** The message has been persisted on our servers. The user can close the app knowing the message will not be lost.\n2. **Delivered (double gray checkmarks):** The message has reached the recipient's device. They may not have seen it yet, but it is on their phone.\n3. **Read (double blue checkmarks):** The recipient has opened the conversation and viewed the message.\n\nNetworks fail, devices go offline mid-delivery, and the same message can be sent twice due to retries, so getting these transitions right takes care.\n\n### The Delivery Flow\n\nThe sequence below maps each checkmark transition to the exact server event that triggers it, showing how a single \"Hello!\" moves from the sender's tap through persistence, delivery, and the final read acknowledgment:\n\n\n```mermaid\nsequenceDiagram\n participant A as User A\n participant S as Server\n participant B as User B\n\n A->>S: Send \"Hello!\"\n Note over S: Store in database\n S-->>A: ACK (message_id: 123)\n Note over A: Show ✓ (Sent)\n\n S->>B: Push \"Hello!\"\n B-->>S: ACK received\n Note over S: Update status\n S-->>A: Status: delivered\n Note over A: Show ✓✓ (Delivered)\n\n Note over B: User opens conversation\n B->>S: Mark 123 as read\n Note over S: Update status\n S-->>A: Status: read\n Note over A: Show ✓✓ (Read/Blue)\n```\n\n\n### Ensuring At-Least-Once Delivery\n\nThe cardinal rule of messaging: **messages must never be lost.** A user who sees the \"sent\" checkmark should be confident that their message will eventually reach its destination, even if networks fail, servers crash, or the recipient's phone runs out of battery.\n\nAchieving this requires a combination of two techniques: the client retries aggressively, and the server deduplicates.\n\n#### Client-Side Retry with Idempotency\n\nReliable messaging relies on one idea: if the server can detect duplicate messages, the client can retry as many times as needed without the message appearing twice.\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n participant DB as Database\n\n Note over C: Generate client_message_id: \"abc-123\"\n\n C->>S: Send (client_id: abc-123)\n Note over C,S: Network timeout - no response\n\n C->>S: Retry (client_id: abc-123)\n S->>DB: Does abc-123 exist?\n DB-->>S: No\n S->>DB: Store message\n S-->>C: ACK (stored as msg_456)\n\n C->>S: Retry again (client_id: abc-123)\n S->>DB: Does abc-123 exist?\n DB-->>S: Yes, already stored as msg_456\n S-->>C: ACK (already stored as msg_456)\n Note over S: Deduplicated!\n```\n\n\nHere is how this works in practice:\n\n1. Before sending, the client generates a unique `client_message_id` (typically a UUID). This ID is the message's fingerprint.\n2. The client sends the message to the server and starts a timer.\n3. If no acknowledgment arrives within a timeout (say, 5 seconds), the client assumes something went wrong and retries with the exact same `client_message_id`.\n4. When the server receives a message, it checks: \"Have I seen this `client_message_id` before?\"\n5. If yes, it is a duplicate. The server returns success without storing again.\n6. If no, it is a new message. The server stores it and returns success.\n\nThis is **idempotent delivery**: the same send can be retried any number of times, and the server still stores and delivers it once.\n\n#### Server-Side Persistence Before Acknowledgment\n\nThere is one more critical rule for reliable messaging: **never acknowledge a message until it is persisted to durable storage.**\n\n\n```mermaid\nflowchart LR\n A[Receive Message]:::primary --> B[Write to Database]:::secondary\n B --> C{Write Success?}:::orange\n C -->|Yes| D[Send ACK to Client]:::green\n C -->|No| E[Return Error]:::red\n E --> F[Client Retries]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nIf the server crashes between receiving a message and persisting it, the message is lost. By only sending ACK after persistence, we guarantee that any acknowledged message is safely stored.\n\n### Handling Out-of-Order Messages\n\nA single connection preserves order, since TCP delivers bytes in sequence. Messages can still arrive out of order for other reasons: a message that timed out and was retried, messages sent from a user's different devices, or messages that travel through different servers and queues on the way to the recipient.\n\n\n```mermaid\nflowchart LR\n subgraph Sent[\"Sent Order\"]\n S1[\"Msg 1: Hello\"]:::primary\n S2[\"Msg 2: How are you?\"]:::primary\n end\n\n subgraph Network[\"Network Chaos\"]\n N[Retries and async
delivery paths]:::orange\n end\n\n subgraph Received[\"Received Order\"]\n R1[\"Msg 2: How are you?\"]:::red\n R2[\"Msg 1: Hello\"]:::red\n end\n\n subgraph Display[\"Correct Display\"]\n D1[\"Msg 1: Hello\"]:::green\n D2[\"Msg 2: How are you?\"]:::green\n end\n\n S1 & S2 --> N --> R1 & R2\n R1 & R2 --> SORT[Sort by
server order]:::secondary\n SORT --> D1 & D2\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe order is established on the server, not the client. The client's clock cannot be trusted (it may be wrong or deliberately set), so we never order by the client timestamp.\n\n1. **Server-assigned ordering:** When the Message Service persists a message, it assigns a server timestamp and a TimeUUID. That TimeUUID is the `message_id` we use as the clustering key in the messages table, so the database stores messages in this exact order. This is the authoritative order for a conversation.\n2. **Per-conversation sequence numbers (when you need gap detection):** A timestamp tells you the order but not whether a message is missing. If the client needs to detect gaps (\"I have messages 1, 2, and 4, so 3 is missing\"), assign a monotonic per-conversation sequence number. Generating one requires routing all of a conversation's messages through a single writer (the partition leader for that conversation), which is why this is reserved for cases that need it rather than applied everywhere.\n3. **Client-side reordering:** The client buffers incoming messages and sorts them by the server order before display, so a late or retried message slots into its correct position instead of appearing at the bottom.\n\n---\n\n## 6.3 Presence System (Online/Offline Status)\n\nThe green \"online\" dot and \"last seen at 3:45 PM\" text seem like simple features. But think about what they require at scale: tracking 50 million concurrent users, notifying their contacts when status changes, and doing it all without overwhelming the system.\n\nPresence is a trade-off between accuracy and efficiency, and the hard part is doing it at scale.\n\n### The Challenges\n\nThe core challenges with presence are:\n\n- **Constant churn:** Users open and close apps constantly. Their phones switch between WiFi, cellular, and airplane mode. Status changes happen all the time.\n- **Fanout volume:** If User A has 500 contacts and comes online, do we notify all 500? What if 1000 users come online in the same second?\n- **Accuracy vs. efficiency:** Users want accurate presence, but broadcasting every change would overwhelm the system.\n\n### Approach: Heartbeat-Based Presence with Lazy Queries\n\nThe practical solution is a combination of heartbeats for tracking and lazy queries for display.\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant CS as Chat Server\n participant R as Redis\n\n C->>CS: Connect + \"I'm online\"\n CS->>R: SET presence:user123 \"online\"
EXPIRE 30 seconds\n\n loop Every 10 seconds\n C->>CS: Heartbeat (I'm still here)\n CS->>R: SET presence:user123 \"online\"
EXPIRE 30 seconds\n end\n\n Note over C: User closes app\n Note over C: No more heartbeats\n\n Note over R: 30 seconds pass...\n Note over R: Key expires automatically\n Note over R: User is now \"offline\"\n```\n\n\n#### How It Works\n\nThe mechanism is simple:\n\n1. **Connection:** When a user connects, the chat server sets a key in Redis: `presence:user_123 = online` with a TTL of 30 seconds.\n2. **Heartbeat:** Every 10 seconds, the client sends a heartbeat over the WebSocket. The server refreshes the TTL, resetting it to 30 seconds.\n3. **Disconnect:** If heartbeats stop (the user closed the app, lost network, or the phone died), the key expires automatically after 30 seconds.\n4. **Query:** When User B opens a chat with User A, the app queries: \"What is the presence of User A?\" Redis returns the value if the key exists, or nothing if it has expired.\n\nThe 30-second TTL is a deliberate choice. It means users appear offline within 30 seconds of going offline, which is acceptable for casual chat. If you needed faster detection (for a stock trading app, say), you could reduce the TTL and heartbeat interval, at the cost of more traffic.\n\n#### Optimizing Presence Fanout\n\nBroadcasting every status change to all contacts does not scale, so instead we query presence only when it is needed:\n\n\n```mermaid\nflowchart LR\n subgraph Trigger[\"User Opens Chat\"]\n A[\"User A opens chat
with User B\"]:::primary\n end\n\n subgraph Query[\"Lazy Query\"]\n Q[\"GET presence:user_b\"]:::secondary\n end\n\n subgraph Redis[\"Redis\"]\n R[(\"user_b: online
TTL: 25s remaining\")]:::teal\n end\n\n subgraph UI[\"Update UI\"]\n D[\"Show: User B is online\"]:::green\n end\n\n A --> Q --> R --> D\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nWhen User A opens a chat with User B:\n\n1. Query Redis for User B's presence\n2. Display result in UI\n3. Subscribe to presence updates (via Redis pub/sub) for real-time changes while chat is open\n4. Unsubscribe when chat is closed\n\nThis drastically reduces presence traffic. We only track presence for users the client is actively viewing.\n\n### Last Seen Timestamp\n\nInstead of binary online/offline, many apps show \"last seen at [time]\":\n\n1. Update `last_seen` timestamp on every meaningful user action\n2. When queried, return the timestamp\n3. Client displays relative time (\"last seen 5 minutes ago\")\n\nThis provides useful information without the complexity of real-time presence. A common variation is to compute live \"online\" status only for a conversation the user currently has open, and fall back to \"last seen\" everywhere else.\n\n---\n\n## 6.4 Scaling Chat Servers\n\nChat servers are stateful: each one holds specific users' WebSocket connections, so a user's messages must reach the exact server they are connected to. That makes them harder to scale than stateless API servers. Let's walk through how to handle it.\n\n### Connection Limits\n\nA well-tuned server with proper kernel configuration can handle 50,000 to 100,000 concurrent WebSocket connections, bounded by file descriptors, memory, and CPU.\n\nAs estimated earlier, covering 50 million average (and ~100 million peak) connections at 50,000 each puts the fleet on the order of 1,000 to 2,000 servers, with headroom so a spike or the loss of a few servers does not exhaust capacity.\n\n\n```mermaid\nflowchart TB\n subgraph Users[\"50 Million Users\"]\n U[\"All Users\"]:::primary\n end\n\n subgraph LB[\"Load Balancing Layer\"]\n L[\"Consistent Hashing
by user_id\"]:::orange\n end\n\n subgraph Servers[\"1,000 Chat Servers\"]\n S1[\"Server 1
50K connections\"]:::secondary\n S2[\"Server 2
50K connections\"]:::secondary\n S3[\"Server 3
50K connections\"]:::secondary\n S1000[\"Server 1000
50K connections\"]:::secondary\n end\n\n U --> L\n L --> S1 & S2 & S3 & S1000\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef secondary fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n### Sticky Sessions and Connection Affinity\n\nBecause a user's connection is pinned to one server until they disconnect, the load balancer must keep routing that user to the same place. Common options:\n\n1. **Consistent hashing by user_id:** Same user always routes to the same server (until server list changes)\n2. **Connection tracking:** Load balancer remembers which server each connection went to\n3. **Client-side server assignment:** Server tells client which specific server to connect to on subsequent attempts\n\n### Handling Server Failures\n\nWhat happens when a chat server crashes? With 50,000 users per server, a crash is a significant event.\n\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S1 as Server 1\n participant LB as Load Balancer\n participant S2 as Server 2\n participant SS as Session Service\n participant MQ as Message Queue\n\n C->>S1: Connected and chatting...\n Note over S1: Server crashes!\n\n C--xS1: Connection lost\n Note over C: Detect disconnect
(no heartbeat response)\n\n C->>LB: Reconnect attempt\n LB->>S2: Route to healthy server\n C->>S2: Establish new connection\n\n S2->>SS: Update: User now on Server 2\n S2->>MQ: Fetch pending messages\n MQ-->>S2: 3 messages queued during crash\n S2->>C: Deliver missed messages\n\n Note over C: Back online, no messages lost\n```\n\n\nThe recovery flow:\n\n1. **Detection:** Client detects disconnection (heartbeat timeout, connection close event)\n2. **Reconnection:** Client connects to load balancer, which routes to a healthy server\n3. **State Recovery:** New server registers the connection in Session Service\n4. **Message Recovery:** Pending messages are fetched from the queue\n5. **Resume:** Normal operation continues\n\nMessage persistence (in the database and queue) is separate from connection state. Even if a server crashes mid-delivery, the message is safe and will be delivered on reconnection.\n\n### Graceful Shutdown\n\nProduction systems need regular maintenance: OS patches, code deployments, hardware replacements. Graceful shutdown minimizes user impact:\n\n1. **Mark as draining:** Remove server from load balancer pool\n2. **Stop new connections:** Reject any new WebSocket handshakes\n3. **Notify clients:** Send a \"please reconnect elsewhere\" message\n4. **Wait for drain:** Give clients time to gracefully disconnect (typically 30-60 seconds)\n5. **Force close:** Terminate any remaining connections\n6. **Shutdown:** Server can now safely restart\n\nMost clients will reconnect to other servers during the drain period, making the maintenance nearly invisible to users.\n\n---\n\n# 7. Follow-ups\n\nWith the core design in place, a few extensions commonly come up as follow-up questions. Two worth being ready for are syncing messages across a user's devices and end-to-end encryption.\n\n## 7.1 Message Synchronization Across Devices\n\nModern users expect their messages on every device: phone, tablet, laptop, web browser. When they read a message on their phone, it should show as read on their laptop too. This is multi-device sync.\n\n### The Challenge\n\nA single incoming message must reach all of a user's connected devices immediately, and queue for the ones that are offline, so the state stays consistent regardless of which device the user picks up next.\n\n\n```mermaid\nflowchart LR\n subgraph UserA[\"User A's Devices\"]\n Phone[\"Phone
Online\"]:::green\n Tablet[\"Tablet
Offline\"]:::red\n Web[\"Web Browser
Online\"]:::green\n end\n\n subgraph Server[\"Message Arrives\"]\n MSG[\"New message
from User B\"]:::primary\n end\n\n MSG --> Phone\n MSG --> Web\n MSG -.->|\"Queue for later\"| Tablet\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nWhen a message arrives for User A, we need to:\n\n1. Push to all online devices immediately\n2. Queue for offline devices\n3. Sync read/delivered status across all devices\n\n### Hybrid Sync Strategy\n\nThe best approach combines real-time push with catch-up pull:\n\n#### Real-time Push (Online Devices)\n\nWhen User A has multiple devices connected, the Session Service tracks all of them:\n\n\n```shell\nuser_a -> [chat_server_1 (phone), chat_server_3 (web)]\n```\n\n\nWhen a message arrives:\n\n1. Session Service returns all device connections\n2. Message is pushed to all connected devices simultaneously\n3. Each device sends independent ACK\n4. \"Delivered\" status is set when ANY device acknowledges\n\n#### Catch-up Pull (Reconnecting Devices)\n\nWhen a device comes online after being offline:\n\n\n```mermaid\nsequenceDiagram\n participant T as Tablet (was offline)\n participant CS as Chat Server\n participant MS as Message Service\n participant DB as Message Store\n\n T->>CS: Connect (last_sync: 2 hours ago)\n CS->>MS: Get messages since last_sync\n MS->>DB: Read messages after marker\n DB-->>MS: 50 messages\n MS-->>CS: 50 new messages\n CS->>T: Here are 50 messages\n T-->>CS: ACK all received\n CS->>MS: Advance device marker\n```\n\n\nThe device sends its last sync timestamp when connecting. The server fetches all messages since then and delivers them in bulk. This ensures no messages are ever missed, regardless of how long the device was offline.\n\n### Read Status Synchronization\n\nWhen User A reads a message on their phone:\n\n\n```mermaid\nsequenceDiagram\n participant P as Phone\n participant S as Server\n participant W as Web Browser\n participant T as Tablet\n\n Note over P: User reads message\n P->>S: Mark message 456 as read\n S->>S: Update database\n\n par Push to sender\n S->>S: Notify User B: message read\n and Sync to other devices\n S->>W: Sync: message 456 read\n S->>T: Sync: message 456 read\n end\n\n Note over W: UI updates: message shown as read\n Note over T: UI updates: message shown as read\n```\n\n\nAll of User A's devices see the same read status. The sender (User B) also gets notified that the message was read.\n\n---\n\n## 7.2 End-to-End Encryption (Conceptual)\n\nEnd-to-end encryption (E2EE) ensures that only the sender and recipient can read messages. Even the service provider (WhatsApp, Signal, etc.) cannot decrypt message content.\n\n### How It Works (Signal Protocol)\n\nMost modern messaging apps use the Signal Protocol or something similar:\n\n\n```mermaid\nsequenceDiagram\n participant A as User A\n participant S as Server\n participant B as User B\n\n Note over A: Generate key pairs
(identity + prekeys)\n Note over B: Generate key pairs
(identity + prekeys)\n\n A->>S: Register public keys + prekeys\n B->>S: Register public keys + prekeys\n\n Note over A: Want to message User B\n A->>S: Get B's prekey bundle\n S-->>A: B's public keys\n\n Note over A: Key agreement (X3DH)
derive shared session key\n Note over A: Encrypt \"Hello!\"
with session key\n A->>S: Send encrypted message\n\n Note over S: I can store this, but
I cannot read it!\n\n S->>B: Forward encrypted message\n Note over B: Derive same session key
decrypt the message\n Note over B: \"Hello!\"\n```\n\n\nThe basic flow:\n\n1. **Key Generation:** Each user's device generates a set of long-term and short-term (prekey) key pairs\n2. **Key Registration:** The public keys and prekeys are uploaded to the server\n3. **Key Agreement:** To start a conversation, the sender fetches the recipient's prekey bundle and runs a key agreement (X3DH) to derive a shared secret. Both sides can compute the same secret without ever transmitting it.\n4. **Encryption:** The message is encrypted with a symmetric session key derived from that shared secret. The Double Ratchet algorithm advances the keys for each message, so a compromised key does not expose past or future messages.\n5. **Transmission:** The encrypted message travels through servers, which can store and route it but cannot decrypt it\n6. **Decryption:** The recipient derives the same session key and decrypts the message\n\n### What the Server Can and Cannot Do\n\nWith E2EE in place, the server's role is limited to routing and storage. The diagram separates what the server retains the ability to do from what it permanently loses access to once encryption is in force:\n\n\n```mermaid\nflowchart LR\n\n subgraph Cannot[\"Server CANNOT\"]\n X1[\"Read message
content\"]:::red\n X2[\"Decrypt any
messages\"]:::red\n X3[\"Provide content
to third parties\"]:::red\n X4[\"Search message
content\"]:::red\n end\n\n subgraph Can[\"Server CAN\"]\n C1[\"Route encrypted
messages\"]:::green\n C2[\"Store encrypted
data\"]:::green\n C3[\"Track delivery
status\"]:::green\n C4[\"Send push
notifications\"]:::green\n end\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n### Trade-offs of E2E Encryption\n\n**Benefits:** E2EE provides strong privacy protection, makes users trust the system more, and helps with regulatory compliance in some regions.\n\n#### Challenges\n\n- **Multi-device complexity:** Each device has its own key pair. Syncing messages across devices requires encrypting for each device separately.\n- **Key changes:** If a user reinstalls the app or gets a new phone, their keys change. The system must handle this securely without enabling man-in-the-middle attacks.\n- **Limited server features:** Search, spam detection, and content moderation become difficult or impossible when the server can't read content.\n- **Backup challenges:** If users back up their messages, the backup is also encrypted. Losing the key means losing access to message history.\n\nFor an interview, it's sufficient to mention that E2EE is important for privacy and explain the high-level concept. The cryptographic details (perfect forward secrecy, double ratchet algorithm, etc.) are typically out of scope unless the interviewer specifically asks.\n\n---\n\n# Quiz","pageType":"system-design-interviews"}

Get Premium