Service Mesh
Practice this topic in a realistic system design interview
Service-to-service communication becomes hard to manage once a system has many independently deployed services. Every call needs timeouts, retries, load balancing, encryption, and metrics, and building all of that into every codebase leads to duplicated work and uneven behavior.
A service mesh moves that networking work out of the applications and into the infrastructure. Applications still make normal network calls, and the mesh handles the rest.
This chapter covers what a service mesh is, what problems it solves in a microservices architecture, how its data plane and control plane work together, and the trade-offs that come with adding one.
1. The Problem with Service-to-Service Communication
Suppose we have a system with fifty microservices that constantly communicate over the network. The order service calls payments, inventory, and shipping. The payment service calls a fraud service, and the fraud service calls a user service.
Every one of these network calls needs many of the same features to work reliably.
| Feature | Why Every Call Needs It |
|---|---|
| Timeouts | A slow service should not block the caller indefinitely |
| Retries | Temporary failures should not fail the whole request |
| Load balancing | Requests should be spread across instances of the target service |
| Encryption | Internal traffic should not be readable or changeable by someone who should not have access |
| Metrics and tracing | Teams need to see which calls are slow or failing |
None of this is business logic. But every service needs these capabilities, and they need to work consistently.
2. The Library Approach and Its Limits
Traditionally, teams handled these concerns using libraries. Each service includes a networking library that provides retries, timeouts, circuit breakers, and metrics. Netflix, for example, built a well-known set of Java libraries for this purpose.
This works well when every service uses the same language. But suppose the order service is written in Java, the payment service in Go, and the recommendation service in Python.
Now the same networking features need to exist in three languages, and they may behave slightly differently in each one.
Upgrades also become difficult. If the security team wants to change how encryption works, every service team may need to update its library, rebuild the application, and redeploy it. Across fifty services, that can take a long time. And there is no guarantee every team will configure the library in exactly the same way.
So instead of putting this networking logic inside every application, can we move it into the infrastructure?
3. What Is a Service Mesh?
That is what a service mesh does. A service mesh is a dedicated infrastructure layer for handling communication between services.
Instead of embedding networking logic inside each application, the mesh places a proxy next to the service. Traffic entering or leaving the service passes through this proxy.
The proxy can handle retries, timeouts, load balancing, encryption, and metrics. The application itself makes a normal network call, while the proxy handles these concerns on its behalf.
Because the same infrastructure is used across the system, services get consistent networking behavior regardless of the language they are written in. Popular service meshes include Istio, Linkerd, and Consul.
A service mesh has two main parts:
- The data plane carries the actual traffic.
- The control plane tells the data plane how that traffic should be handled.