Premium Video

This video is available to premium subscribers only

Unlock Full Access

A heartbeat is a small message that a node sends at regular intervals to let other nodes know it's still running. Distributed systems use heartbeats to detect failed nodes, so they can stop sending them work and start recovery.

The hard part is choosing how often to send heartbeats and how long to wait before acting, so the system catches real failures without raising false alarms.

In this chapter, we'll learn what heartbeats are, how distributed systems use them to detect failed nodes, and how to choose intervals and timeouts that catch real failures without raising false alarms.

A heartbeat is a small message that a node sends at regular intervals to let other nodes know it's still running.

In a distributed system, a node might send a heartbeat every second to a monitor, a coordinator, or other nodes in the cluster. The receiver keeps track of when it last heard from each node.

In other words, heartbeats help the receiver answer one question: have I heard from this node recently enough to keep trusting it?

Let's walk through a simple example. Suppose we have a monitor and three worker nodes. Each worker sends a heartbeat to the monitor every second, and the monitor records when it last received a heartbeat from each worker.

At regular intervals, the monitor checks these timestamps.

Worker 1 last sent a heartbeat half a second ago. Worker 2 last reported 800 milliseconds ago. But worker 3 hasn't sent a heartbeat in five seconds.

Now, suppose the monitor has a timeout of three seconds. Since worker 3 has been silent for longer than that, the monitor marks it as failed.

Based on this decision, the system can stop sending new work to that node, reassign its existing tasks, or alert an operator.

There are two common ways to collect heartbeats.

Push Model

In the push model, each node sends heartbeats at regular intervals, and the monitor simply listens for them. This is the approach we just discussed.

It's simple, and the monitor doesn't need to know every node's address in advance. The weak point is the monitor itself. If it is overloaded or unreachable, healthy nodes can look like they have failed.

Pull Model

In the pull model, the monitor actively checks whether each node is healthy. It sends a ping or calls a health check endpoint, and the node responds.

Load balancers commonly use this approach. They periodically send requests to a health check endpoint on each server. If a server fails to respond correctly several times in a row, the load balancer removes it from rotation.

The pull model gives the monitor more control over when checks happen. However, it also means the monitor has to contact every node, which can become expensive in very large clusters.

Both approaches are widely used, and some systems combine them.

Every heartbeat system has two important settings:

The timeout should usually be several times longer than the heartbeat interval so it's not too sensitive.

For example, with a heartbeat interval of one second and a timeout of five seconds, a node would have to remain silent for about five seconds before being marked as failed. This way, a single dropped packet or a brief network delay won't cause a false alarm.

Many systems express the timeout as a failure threshold: the number of missed heartbeats before the system acts.

Choosing the right values involves a trade-off.

A shorter timeout helps detect failures quickly. If a node crashes, the system can notice within a second or two and start recovering. However, shorter timeouts also increase the risk of false positives. A healthy node that's temporarily slow might be marked as failed.

A longer timeout is more forgiving and reduces false alarms. But if a node actually crashes, the system takes longer to detect the failure and may continue sending work to that node in the meantime.

The heartbeat interval has a similar trade-off. Sending heartbeats more frequently can help detect failures faster, but it also increases network traffic and processing overhead. With thousands of nodes, even small heartbeat messages can add up.

There's no single configuration that works best for every system. It depends on how quickly failures need to be detected and how much damage a false alarm could cause.

Now, let's look at why a healthy node might stop sending heartbeats.

In all these cases, the node hasn't actually crashed. It might resume normal operation within a few seconds. But from the monitor's perspective, there's no way to tell the difference between these temporary delays and a real failure.

And that's the main limitation of heartbeats. A missing heartbeat only tells us that we haven't heard from a node. It doesn't tell us why.

So heartbeat-based failure detection is really about deciding when to suspect that a node has failed, rather than knowing for certain that it's dead.

One way to reduce false positives is to introduce an intermediate state. Instead of marking a node as failed immediately, the system first marks it as suspected.

This helps the system avoid unnecessary failovers caused by temporary network issues.

If a failed node starts sending heartbeats again later, it's safer not to trust it right away. Many systems move it to a recovering state and run checks, such as catching up on missed data, before sending it full traffic again.

So far, we've mostly looked at nodes sending heartbeats to a central monitor. But how well does this approach scale?

Central Monitor

A central monitor is simple to set up, but it can become a single point of failure. And with thousands of nodes, it has to process a large number of heartbeat messages.

All-to-All Heartbeats

Another option is all-to-all heartbeats, where every node sends heartbeats to every other node. This removes the need for a central monitor, but the number of messages grows quadratically with the number of nodes. With a thousand nodes, that's nearly a million messages per interval.

Gossip

A more scalable approach is to use heartbeats with gossip.

Over time, heartbeat information spreads throughout the cluster. If a node's counter stops increasing for long enough, other nodes begin to suspect that it has failed.

The advantage is that each node only needs to contact a few peers per round, rather than every node in the cluster. The trade-off is that different nodes may briefly have different views of which nodes are alive.

A heartbeat doesn't have to be just an empty message. Since nodes are already sending heartbeats regularly, they can also include useful information about their current state.

Many heartbeats also include a sequence number that increases with every message. It lets the receiver detect heartbeats that were missed, duplicated, or arrived out of order.

Keep the message small, though. With thousands of nodes, extra fields add up quickly.

There's an important limitation with heartbeats that we haven't discussed yet. A node can keep sending heartbeats even when it's unable to do any useful work.

For example, suppose heartbeats are sent from a separate background thread. Meanwhile, the main processing thread gets stuck in an infinite loop or waits indefinitely for a database response. The heartbeats keep arriving, so the node appears healthy, even though it isn't processing any requests.

Kafka Consumers

Kafka consumers are a good example of how systems handle this problem. A consumer sends heartbeats in the background to show that it's still part of the consumer group. But Kafka also tracks how long it's been since the consumer last polled for messages. If that gap exceeds a configured limit, Kafka treats the consumer as unresponsive and removes it from the group, even though its heartbeats are still arriving.

Kubernetes Probes

Kubernetes makes a similar distinction with health probes.

The key takeaway is that receiving heartbeats doesn't necessarily mean a node is healthy. A good health check should tell us whether the node can actually do its job, not just whether the process is running.

Let's bring everything together with a few practical guidelines.

Why the Last Rule Matters

Suppose replicas stop hearing from a database primary because of a network partition, and they promote a new primary. The old primary is still running and some clients can still reach it. Now two nodes accept writes, a situation called split-brain.

So missed heartbeats should only start the process. Before promoting a new primary, the system should get a majority of nodes to agree, and use fencing so the old primary can no longer write.