In a distributed system, a node can stop responding for many reasons. It may have crashed, it may be overloaded, or the network path to it may be broken. The system still needs a way to notice. A heartbeat is the usual approach: a small message sent on a schedule that says, in effect, "I am still here."

Distributed systems use heartbeats to detect when nodes, services, workers, or connections may have failed. They are everywhere: databases, queues, Kubernetes, service discovery, leader election, and WebSocket connections all use some version of them.

The most important thing to remember is this: a missing heartbeat does not prove a node is dead. It only means the watcher did not receive a message in time.

Act too quickly, and you may trigger unnecessary failovers or even split-brain, where two nodes both think they own the same job. Act too slowly, and real failures take longer to recover from. Tuning that balance is most of the work.

This chapter covers how heartbeats detect failures and how to tune them.

Heartbeats help answer one question:

Have I heard from this component recently enough to keep trusting it?

They are used to notice possible problems: a crashed node, a stopped process, a broken network path, a stuck worker, a disconnected client, a leader that cannot be reached, or a replica that has fallen behind.

The careful word is suspect. A heartbeat system should usually move a component through states instead of jumping straight from healthy to dead.

This gives the system room to handle short network hiccups without overreacting.

A basic heartbeat system has two roles:

A heartbeat message is usually small. At minimum, it carries a node ID so the watcher knows who sent it. It may also include a timestamp for rough freshness checks and debugging.

A sequence number helps detect missed, duplicated, or reordered heartbeats. Some systems also include the sender's current role (leader, follower, worker, standby), a short health summary like CPU and queue depth, or a replication position that shows whether a replica is still keeping up.

Keep the message small unless the extra data is used for a real decision. In large clusters, even small messages add up quickly.

There are several heartbeat models.

Push Heartbeats

Nodes send heartbeats to a monitor or coordinator.

Push is simple and efficient when many nodes report to one control plane. Kubernetes nodes reporting to the API server are an example of this pattern.

The weakness is the central watcher. If the watcher is overloaded or unreachable, many healthy nodes may look unhealthy.

Pull Heartbeats

The watcher periodically checks each node.

Pull is common for load balancers and service health checks. It verifies that the node can respond to a specific kind of request.

The weakness is scale. A monitor probing thousands of nodes too often can become a bottleneck.

Gossip Heartbeats

Nodes exchange health information with peers instead of reporting only to one monitor.

Gossip scales well because failure information spreads through the cluster gradually. Systems such as Cassandra and Consul use gossip-style membership and failure detection.

The trade-off is that detection is not instant everywhere. Different nodes may briefly have different views of who is alive.

Heartbeat behavior is mostly controlled by three settings:

Example:

A common starting point is to declare suspicion after 2-5 missed heartbeats. The right value depends on the network, workload, and cost of being wrong.

Fast Detection vs. False Alarms

Every choice here is a trade. A short interval and low threshold give fast failover but produce false alarms during latency spikes or garbage collection pauses.

A long interval and high threshold avoid false alarms at the cost of slower recovery from real failures.

An adaptive timeout that adjusts to observed latency fits changing networks better, but is more work to build. A multi-stage suspicion model (healthy -> suspect -> unhealthy) avoids many bad decisions, at the cost of more states to reason about.

There is no perfect timeout. The system is always acting on incomplete evidence. You are choosing how quickly it should act and how much risk of a false alarm you can tolerate.

Missed heartbeats do not always mean crashes. The process may truly be gone, but many other things can make a heartbeat arrive late or not arrive at all.

A network partition can stop heartbeats from reaching the watcher even though the sender is healthy. A maxed-out CPU can leave the process alive but unable to schedule the heartbeat work.

A garbage collection pause, where the runtime briefly stops application work to clean memory, can freeze the process long enough to miss a deadline. A disk stall can do the same. Packet loss and congestion can delay or drop heartbeats while they are crossing the network.

Even the watcher side matters. An overloaded monitor may receive heartbeats but process them too late, and local clock or timer issues can throw off scheduling on either end.

This is why missed heartbeats should be treated as warning signs, not facts.

The response should match the risk. Removing a backend from a load balancer after missed heartbeats is usually fine. Promoting a new database primary requires stronger safeguards, such as majority agreement and fencing, which blocks the old primary from writing.

Heartbeats are often used to trigger failover. That makes them useful, but also dangerous.

Consider replicas that stop receiving heartbeats from the primary. They may conclude the primary is dead and promote a new primary. But the old primary may still be alive and reachable by some clients.

This is how heartbeat-based failure detection can contribute to split-brain if failover is not protected.

Good systems do not promote a new leader only because heartbeats stopped.

They combine heartbeats with stronger safety checks:

Heartbeats can tell you when to investigate or start an election. They should not be the only proof that it is safe to take exclusive ownership.

A heartbeat system should define actions for each state.

For stateless services, recovery may be simple: remove the instance from rotation and let the platform restart it.

For stateful systems, recovery needs more care. Check replication lag before promoting a replica. Make sure the old leader is fenced before accepting writes elsewhere.

Rebuild or resync replicas that fell behind before they serve reads. Avoid sending traffic back to a node simply because one heartbeat returned. One message is not strong evidence that the node is ready again.

Heartbeats and health checks are related, but not identical.

A process can be alive but not ready. It can send heartbeats while its database connection pool is exhausted. It can pass a shallow health check while failing real user requests.

Use the cheapest signal that is safe for the decision being made. Restarting a process, routing traffic to it, and promoting it to leader are different decisions, so they need different levels of confidence.

Heartbeats show up almost everywhere distributed systems run.

In Kubernetes, kubelets report node status to the control plane, while liveness and readiness probes decide whether each pod should run and receive traffic. In Kafka consumer groups, consumers send heartbeats to the coordinator to keep their group membership; missing them triggers a partition reassignment.

In Raft-based systems, the leader sends heartbeats to followers to show that it is still the leader and to prevent unnecessary elections. ZooKeeper-style sessions are kept alive by client heartbeats, and session expiry removes temporary state.

Load balancers run periodic health checks to remove unhealthy backends from rotation. Storage systems use heartbeats from data nodes to report whether nodes are alive, how much disk is used, and which data blocks they hold. Even WebSockets rely on ping/pong messages to detect broken long-lived connections.

The details differ, but the pattern is the same: regular communication helps the system decide whether it can still trust a component.

A few habits separate a reliable heartbeat system from a noisy one.

Always pass through a suspect state before declaring failure, so one missed message does not trigger a recovery action. Tune intervals from observed latency in your environment, since production networks vary by region, workload, and time of day.

Add jitter, meaning a small random delay, so nodes do not all send heartbeats at the same instant and overwhelm the monitor. Include sequence numbers to detect missed, duplicated, or reordered messages.

Beyond the message itself, separate liveness from readiness so traffic is never sent to a process that is alive but not yet able to serve. Protect failover with majority agreement or fencing so missed heartbeats cannot turn into split-brain.

Monitor heartbeat lag and false-alarm rates to learn whether detection is too aggressive. Rate-limit recovery actions so a wave of suspicion does not turn into a restart or failover storm.

A missed heartbeat signals a possible problem, not a confirmed one. Treating it as certainty leads to most heartbeat-related failures.

Heartbeats are small messages used to check whether a component is still communicating. A received heartbeat is evidence that a component was recently reachable. A missed heartbeat is a warning sign, not proof of failure.

Short timeouts detect failures faster but raise false alarms. Long timeouts reduce false alarms but slow recovery. Stateful failover needs stronger protection than heartbeats alone.

Use heartbeats to create timely suspicion. Then use majority agreement, fencing, leases, readiness checks, and recovery rules to decide what action is actually safe.