Replication Deep Dive for System Design Interviews
Replication Deep Dive for System Design Interviews
Replication is the foundation of distributed systems. When your interviewer asks "How do you ensure high availability?" or "What happens if a database server fails?", replication is almost always part of the answer.
Yet many candidates only scratch the surface: "We replicate the data to multiple servers." That is not enough. Interviewers want to know the replication topology, synchronous vs asynchronous trade-offs, how you handle replication lag, what happens during failover, and how you resolve conflicts.
This article provides a deep understanding of replication for system design interviews. We will explore why replication matters, different replication architectures, consistency guarantees, failover mechanisms, and how major databases implement replication.
Here is what we will cover:
- Why Replication Matters
- Single-Leader Replication
- Multi-Leader Replication
- Leaderless Replication
- Synchronous vs Asynchronous Replication
- Replication Lag and Its Problems
- Consistency Guarantees
- Failover and High Availability
- Replication in Practice
- Common Interview Questions
If you're enjoying this newsletter and want to get even more value, consider becoming a [paid subscriber](https://blog.algomaster.io/subscribe).
As a paid subscriber, you'll unlock all premium articles and gain full access to all [premium courses](https://algomaster.io/newsletter/paid/resources) on [algomaster.io](https://algomaster.io).
1. Why Replication Matters
Before diving into implementation details, let us understand why replication is fundamental to distributed systems.
1.1 The Single Server Problem
A single database server is a single point of failure.
Problems with a single server:
1.2 What Replication Provides
Replication maintains copies of the same data on multiple machines.
Benefits of replication:
1.3 The Fundamental Challenge
Replication sounds simple, just copy the data. But keeping replicas in sync is hard.
Key questions replication must answer:
- Which node accepts writes?
- How do writes propagate to replicas?
- What happens during network partitions?
- How do we handle conflicting writes?
- What consistency guarantees can we provide?
Interview insight: "Replication is not just about copying data. It is about defining which node is authoritative, how changes propagate, and what consistency guarantees we provide. The architecture we choose depends on our availability requirements, latency tolerance, and consistency needs."
2. Single-Leader Replication
Single-leader (also called primary-replica or master-slave) is the most common replication architecture.
2.1 How It Works
One node is designated as the leader. All writes go to the leader, which then propagates changes to followers.
2.2 Replication Process
Key components:
2.3 Statement-Based vs Log-Based Replication
Statement-based replication:
Log-based (row-based) replication:
Most modern databases use log-based replication for reliability.
2.4 Advantages and Disadvantages
Advantages:
Disadvantages:
2.5 Read Replicas for Scaling
Single-leader replication excels at scaling reads.
Scaling strategy:
Interview tip: "For our read-heavy workload with a 50:1 read/write ratio, we use a single leader for all writes and 10 read replicas behind a load balancer. This allows us to handle 500,000 reads per second while the leader handles 10,000 writes per second."
3. Multi-Leader Replication
Multi-leader (also called active-active or master-master) allows writes on multiple nodes.
3.1 When to Use Multi-Leader
Primary use case: Multi-datacenter deployment
3.2 Advantages Over Single-Leader
3.3 The Conflict Problem
The biggest challenge with multi-leader is conflicting writes.
3.4 Conflict Resolution Strategies
Strategy 1: Last Write Wins (LWW)
Strategy 2: Merge Values
Strategy 3: Custom Application Logic
Strategy 4: Operational Transformation (OT)
Used by collaborative editors like Google Docs.
3.5 Conflict Avoidance
The best conflict is one that never happens.
Interview insight: "We use multi-leader replication across our three datacenters, but we route each user to their home datacenter for writes. This avoids most conflicts. For the rare case of cross-datacenter conflicts (like a user traveling), we use LWW with logical timestamps."
4. Leaderless Replication
Leaderless (also called Dynamo-style) replication has no designated leader. Any node can accept writes.
4.1 How It Works
Key concepts:
4.2 Quorum Writes and Reads
For consistency, we need: w + r > n
Why w + r > n works:
4.3 Common Quorum Configurations
4.4 Sloppy Quorums and Hinted Handoff
When nodes are unavailable, strict quorums would reject writes. Sloppy quorums provide higher availability.
How hinted handoff works:
- Node 1 is down
- Write goes to Node 4 (not normally responsible for this key)
- Node 4 stores a "hint" that data belongs to Node 1
- When Node 1 recovers, Node 4 sends the data
- Node 4 deletes the hinted data
4.5 Read Repair and Anti-Entropy
Leaderless systems need mechanisms to fix stale replicas.
Read repair:
Anti-entropy process:
Background process that compares replicas and fixes inconsistencies.
4.6 Databases Using Leaderless Replication
Interview tip: "For our shopping cart service, we use Cassandra with quorum reads and writes (w=2, r=2, n=3). This gives us high availability while ensuring customers see their cart correctly. For less critical data like view counts, we use w=1 for faster writes."
5. Synchronous vs Asynchronous Replication
The choice between sync and async replication is one of the most important trade-offs.
5.1 Synchronous Replication
Leader waits for followers to confirm before acknowledging the write.
Properties:
5.2 Asynchronous Replication
Leader acknowledges immediately, replicates in the background.
Properties:
5.3 Semi-Synchronous Replication
Wait for at least one follower, rest are async.
Benefits:
- Guaranteed durability on at least 2 nodes
- Lower latency than full sync
- If sync follower fails, another is promoted to sync
This is the default in many production setups (MySQL semi-sync, PostgreSQL synchronous_standby_names).
5.4 Comparison
5.5 Durability Guarantees
Interview insight: "For our payments service, we use semi-synchronous replication. The leader waits for at least one follower to confirm before acknowledging the write. This gives us strong durability without the latency impact of waiting for all replicas. For our analytics pipeline, we use async replication because occasional data loss is acceptable and we need higher throughput."
6. Replication Lag and Its Problems
Asynchronous replication means followers can fall behind. This creates consistency problems.
6.1 What Is Replication Lag?
Causes of replication lag:
6.2 Reading Your Own Writes
The problem:
Solution: Read-your-writes consistency
Alternative approaches:
- Track LSN of user's last write, read from replica only if caught up
- Always read user's own data from leader
- Client remembers last write timestamp, includes in request
6.3 Monotonic Reads
The problem:
User reads from different replicas with different lag, sees data "going back in time."
Solution: Monotonic reads
Alternative: Track min LSN seen by user, only read from replicas at or past that LSN.
6.4 Consistent Prefix Reads
The problem:
Causally related writes appear in wrong order.
Solution: Consistent prefix reads
- Writes that are causally related must be in same partition
- Or use logical timestamps to order events
- Or read from single replica
6.5 Monitoring Replication Lag
Key metrics to monitor:
Interview tip: "We monitor replication lag and route reads appropriately. For user-facing requests requiring fresh data, we read from the leader or wait for a replica with acceptable lag. For analytics queries where staleness is okay, we read from any replica."
7. Consistency Guarantees
Different applications need different consistency levels. Understanding these is crucial for interviews.
7.1 Consistency Spectrum
7.2 Eventual Consistency
Weakest guarantee: If no new writes, all replicas eventually converge.
7.3 Strong Consistency (Linearizability)
Strongest guarantee: All operations appear to occur instantly at some point between invocation and response.
Properties:
- Behaves like a single copy
- All clients see same order of operations
- Real-time ordering preserved
Cost:
- Higher latency (coordination required)
- Lower availability during partitions
- More complex implementation
7.4 Causal Consistency
Preserves cause-effect relationships without global ordering.
Rules:
- If A causes B, everyone sees A before B
- Concurrent (unrelated) events can be seen in any order
7.5 Choosing Consistency Level
8. Failover and High Availability
What happens when the leader fails? Failover is critical for availability.
8.1 Detecting Failure
Detection methods:
Challenges:
- Network partition vs actual failure
- Slow leader vs dead leader
- False positives cause unnecessary failovers
8.2 Failover Process
Failover steps:
- Detect failure: Monitors identify leader is unresponsive
- Choose new leader: Select follower with most recent data
- Promote follower: Configure as new leader
- Reconfigure replicas: Point other followers to new leader
- Update routing: Direct clients to new leader
8.3 Choosing the New Leader
Selection criteria:
8.4 Split-Brain Problem
The most dangerous failover scenario: two nodes both believe they are leader.
Consequences:
- Both accept writes
- Data diverges
- Conflicts after partition heals
- Potential data loss
Prevention strategies:
8.5 Automatic vs Manual Failover
8.6 Failover Tools
Interview insight: "We use Patroni for PostgreSQL failover with a 30-second detection timeout. When the leader fails, the most up-to-date replica is promoted automatically. We use etcd as the consensus store to prevent split-brain, requiring a majority of nodes to agree on the leader."
9. Replication in Practice
Let us see how major databases implement replication.
9.1 PostgreSQL Streaming Replication
Configuration:
Replication modes:
9.2 MySQL Replication
Replication types:
9.3 MongoDB Replica Sets
Write concern options:
Read preference options:
9.4 Cassandra Replication
Consistency levels:
9.5 Comparison Summary
10. Common Interview Questions
10.1 Design Questions
Q: How would you set up replication for a global e-commerce platform?
Q: Design replication for a real-time chat application.
10.2 Troubleshooting Questions
Q: Replication lag is growing. How do you debug?
Q: How do you handle failover without data loss?
10.3 Quick Reference
Summary
Replication is fundamental to building reliable distributed systems. Here are the key takeaways:
- Understand the trade-offs. Replication involves trade-offs between consistency, availability, and latency. There is no perfect solution, only appropriate solutions for your requirements.
- Know the topologies. Single-leader for simplicity and consistency, multi-leader for global writes, leaderless for highest availability. Choose based on your read/write patterns and geographic distribution.
- Sync vs async is crucial. Synchronous replication ensures durability but adds latency. Asynchronous replication is faster but can lose data on leader failure. Semi-synchronous is often the right balance.
- Replication lag causes problems. Reading your own writes, monotonic reads, and consistent prefix reads are common issues. Design your application to handle lag or enforce consistency where needed.
- Consistency is a spectrum. From eventual to linearizable, different operations need different guarantees. Strong consistency everywhere is expensive and often unnecessary.
- Failover is critical and complex. Detecting failures, choosing new leaders, and preventing split-brain requires careful design. Use battle-tested tools rather than building your own.
- Conflicts are inevitable in multi-leader. If you accept writes on multiple nodes, you must handle conflicts. Last-write-wins, merging, or application logic are your options.
- Monitor replication health. Track lag, connection status, and throughput. Set up alerts before problems become outages.
- Different databases, different models. PostgreSQL, MySQL, MongoDB, and Cassandra all handle replication differently. Know the specifics of your database.
- Design for failure. Assume nodes will fail, networks will partition, and disks will corrupt. Your replication architecture should handle these gracefully.
When discussing replication in interviews, be specific about your choices. Do not just say "we replicate the data." Explain the topology, whether it is synchronous or asynchronous, how you handle lag, what consistency guarantees you provide, and how failover works. This depth demonstrates genuine understanding of distributed systems.
Thank you for reading!
If you found it valuable, hit a like and consider subscribing for more such content every week.
If you have any questions or suggestions, leave a comment.
This post is public so feel free to share it.
P.S. If you're enjoying this newsletter and want to get even more value, consider becoming a [paid subscriber](https://blog.algomaster.io/subscribe).
As a paid subscriber, you'll unlock all premium articles and gain full access to all [premium courses](https://algomaster.io/newsletter/paid/resources) on [algomaster.io](https://algomaster.io).
There are [group discounts](https://blog.algomaster.io/subscribe?group=true), [gift options](https://blog.algomaster.io/subscribe?gift=true), and [referral bonuses](https://blog.algomaster.io/leaderboard) available.
Checkout my [Youtube channel](https://www.youtube.com/@ashishps_1/videos) for more in-depth content.
Follow me on [LinkedIn](https://www.linkedin.com/in/ashishps1/) and [X](https://twitter.com/ashishps_1) to stay updated.
Checkout my [GitHub repositories](https://github.com/ashishps1) for free interview preparation resources.
I hope you have a lovely day!
See you soon,
Ashish
References
- Designing Data-Intensive Applications - Martin Kleppmann's definitive guide to replication and distributed systems
- PostgreSQL Replication Documentation - Official PostgreSQL streaming replication guide
- MySQL Replication - MySQL replication configuration and best practices
- MongoDB Replica Sets - MongoDB official documentation on replica sets
- Dynamo: Amazon's Highly Available Key-value Store - The paper that defined leaderless replication
- Jepsen: Distributed Systems Safety Research - Kyle Kingsbury's analysis of consistency in distributed databases