Design Pub Sub System
Design a Publish-Subscribe (Pub-Sub) system where publishers send messages to topics and subscribers receive messages for the topics they are subscribed to.
A Publish-Subscribe (Pub-Sub) system is a messaging pattern where publishers send messages to topics without knowing who will receive them, and subscribers receive messages by subscribing to those topics.
This decouples senders and receivers, making it easier to build scalable and flexible systems. Popular examples include Kafka, Redis Pub/Sub, and Google Cloud Pub/Sub.
In this chapter, we will explore the low-level design of a simple in-memory pub-sub system.
Lets start by clarifying the requirements:
1. Clarifying Requirements
Before starting the design, it's important to ask thoughtful questions to uncover hidden assumptions and better define the scope of the system.
Here is an example of how a conversation between the candidate and the interviewer might unfold:
Candidate: Should the system support multiple publishers and subscribers for a single topic?
Interviewer: Yes, each topic can have multiple publishers and multiple subscribers. For this design, the publisher can simply be the client / demo using it.
Candidate: Should message delivery be synchronous or asynchronous?
Interviewer: It should be asynchronous. When a publisher sends a message, it should not wait for subscribers to consume it.
Candidate: Do we need to guarantee message ordering for subscribers?
Interviewer: Yes, messages within a topic should be delivered to each subscriber in the order they were published.
Candidate: Should we support different delivery semantics like exactly-once or at-least-once delivery?
Interviewer: For this design, let’s keep it simple and go with a "fire-and-forget" model. That means best-effort delivery without retries, acknowledgments, or delivery guarantees.
Candidate: Can subscribers unsubscribe at any point during runtime?
Interviewer: Yes, subscribers should be able to dynamically subscribe and unsubscribe from topics at any time.
After gathering the details, we can summarize the key system requirements.
1.1 Functional Requirements
- Support creation and management of multiple topics
- Allow multiple publishers to publish messages to a topic
- Allow multiple subscribers to subscribe to one or more topics
- Deliver messages to all active subscribers of a topic in the order they were published
- Ensure non-blocking, asynchronous message delivery
- Follow a "fire-and-forget" delivery model: no retries, or acknowledgments
1.2 Non-Functional Requirements
- Modularity: The system should follow object-oriented principles with clear separation of responsibilities
- Scalability: The system should efficiently support many concurrent publishers and subscribers
- Extensibility: The design should be flexible enough to support future enhancements such as message persistence, retries, or delivery guarantees
- Reliability: While exact delivery guarantees are not required, message ordering and dispatching must remain consistent and predictable within each topic
After the requirements are clear, lets identify the core entities/objects we will have in our system.