In a single process, failures are often direct. A function returns a value, throws an exception, or the process stops. Across a network, the evidence is weaker.

A caller may receive a connection error before sending anything. It may send a complete request and then lose the connection. It may receive an explicit server error, a malformed response, or no response before its deadline. Meanwhile, the remote operation may not have started, may still be running, or may already have completed.

This makes “the call failed” an incomplete diagnosis. Distributed failures have three different aspects:

These aspects do not always agree. A caller can observe a timeout even though the server succeeded, and one caller can observe a healthy service while another is cut off by a network partition.

Understanding failure modes means learning to preserve those distinctions. It is the foundation for making correct decisions when part of a distributed system stops behaving normally.

A service call is not one indivisible action. It is a sequence of stages, and each stage produces different evidence when it fails.

Before a Connection Exists

The name resolver may fail to find an address, or resolution may take too long. The client may then wait for a connection from a saturated pool. If it creates a new connection, the attempt can be refused, time out, or fail during transport-security negotiation.

These signals are not interchangeable.

A direct TCP connection that is immediately refused normally means the destination host responded but no process accepted that connection at the selected address and port. No application request crossed that new connection.

A connect timeout means the client did not establish a connection within its limit. The target may be down, packets may be filtered, the path may be broken, or the response may simply be delayed. Silence provides less information than a refusal.

Name resolution and connection-pool waiting are also separate stages. Some client libraries do not include DNS lookup in their connect timeout, and a request can exhaust its useful time while waiting for a pooled connection without attempting the network at all.

After Request Transmission Begins

Once the client starts writing, failures become more ambiguous. A broken connection can occur after the server has received none, some, or all of the request. Even if the client finishes its local write, the bytes may only have reached a local kernel buffer; that does not prove that the server application read them.

The server can explicitly reject the request with an application response. An HTTP 400, 404, 429, or 503 is not a missing network response—it is a response carrying failure information. That evidence is usually more useful than silence, although the precise business meaning still comes from the API contract.

The connection can also fail while the client reads the response. The client might receive headers but only part of the body, or the server might finish an operation and then crash before returning anything. A syntactically complete response can still fail validation if its body violates the expected schema or contains an impossible value.

A useful error model therefore preserves the phase and signal:

Flattening all of these into service unavailable discards evidence needed for correct handling and diagnosis.

Several failure categories describe what a distributed component does. They overlap with the stages above, but they answer a different question: what behavior did the system exhibit?

Crash Failure

A process or machine stops executing. It no longer accepts new work or responds to existing work.

The simplest theoretical model is crash-stop: once a node crashes, it never returns. Real services more often exhibit crash-recovery behavior. A process restarts, possibly with durable state from before the crash but without its in-memory queues, connections, or partially completed work.

Recovery does not erase the failure. Existing connections are broken, clients may still hold stale endpoint state, and work that was only in memory may be gone.

Omission Failure

A component fails to send or receive something it was expected to handle. A request can disappear before reaching the application, the server can ignore it, or the response can fail to reach the caller.

Reliable transports recover from ordinary packet loss, so applications do not observe every dropped packet as an omission. They observe the cases the transport cannot hide: a failed exchange, an ended connection, or silence beyond the application's time limit.

Timing Failure

A response is correct but arrives outside the period in which it is useful. The service may be slow because of CPU saturation, a long queue, a paused runtime, a slow dependency, congestion, or physical distance.

Distributed applications often cannot distinguish “too slow” from “never coming.” They define a time boundary and treat both as failure after that point.

Response Failure

A component responds, but the response is invalid or incorrect. It may use the wrong schema, return corrupted content, violate an invariant, or report success for work that did not occur.

The broad theoretical category of arbitrary or Byzantine failure includes nodes that behave inconsistently or maliciously. Most backend systems are not built to tolerate fully Byzantine participants, but they still validate message framing, schemas, checksums, authentication, and important business invariants. A fast response is not automatically a correct response.

Partition Failure

Groups of otherwise running nodes lose the ability to communicate reliably with one another. Each group may still serve local clients and communicate internally. The system is split rather than entirely down.

These categories are not mutually exclusive. A crashed proxy can create omissions, which produce timeouts at callers and may make two healthy groups appear partitioned. The taxonomy helps describe behavior; it does not imply that every incident has only one label.

A timeout is a limit on how long an application will wait for a particular event. It is a local policy decision, not a diagnosis of the remote system.

When 500 milliseconds pass without a response, the caller learns only this:

It does not learn that the server is dead, that the network dropped the request, or that the operation failed. The event might arrive at 501 milliseconds. The server might still be queued, or it might have committed the operation and lost the response.

Without timeouts, a slow or unreachable dependency can hold sockets, memory, worker threads, and request slots indefinitely. With timeouts, the caller bounds that resource use and regains control. The uncertainty remains, but it becomes an explicit program outcome.

Timeouts Apply to Different Phases

“The timeout is two seconds” is underspecified because network clients commonly expose several timers.

Pool-acquisition timeout limits how long a request waits for an available connection from the local pool.

Connect timeout limits connection establishment with an endpoint. It may not include endpoint resolution.

Security-handshake timeout bounds protocol negotiation after the underlying connection is made.

Write timeout limits progress while transmitting the request. This matters when the peer or path accepts data too slowly.

Response-header or first-byte timeout limits the wait until the response begins.

Read or inactivity timeout limits how long the client tolerates no response-body progress. In many libraries, receiving another chunk resets this timer.

Overall request timeout limits the entire operation, regardless of progress in individual phases.

The distinction between inactivity and total duration matters. A server that sends one byte periodically can keep resetting an inactivity timeout while taking minutes to finish. An overall deadline still bounds the exchange.

Library names and exact coverage differ. Engineers must verify whether a configured timeout includes DNS, pool waiting, redirects, request upload, and response-body consumption rather than inferring behavior from the option's name.

Timeouts and Deadlines

A relative timeout says, “allow this operation to run for 300 milliseconds starting now.” A deadline says, “this work is no longer useful after this absolute time.”

Deadlines compose better across service chains.

Suppose a gateway has 800 milliseconds to answer a user request. It spends 120 milliseconds authenticating and 180 milliseconds calling the Order service. Only 500 milliseconds remain, and some of that is needed to construct and transmit the final response.

Giving the next dependency a fresh 800-millisecond timeout would let internal work continue after the gateway can no longer use the result. Instead, the downstream allowance must fit within the remaining budget:

Propagating the deadline lets each service reject work that is already too late. It also makes logs from different services describe the same end-to-end time boundary.

Clock interpretation requires care. Within one process, a monotonic clock is appropriate for measuring elapsed duration because wall-clock corrections should not make a timeout run backward. Across processes, protocols often propagate a remaining duration or a carefully defined absolute deadline. The implementation must account for clock uncertainty and transmission time.

Choosing a Timeout

A timeout that is too short creates false timeouts: operations that would have succeeded within an acceptable period are abandoned early. A timeout that is too long allows slow dependencies to consume upstream resources and exceed the user's latency objective.

There is no universal correct value. The choice should reflect:

Measurements should separate warm calls using established connections from cold calls that perform resolution and connection setup. One global timeout applied to every dependency and every operation ignores meaningful differences.

A timeout should be tested under saturation as well as normal traffic. Queueing can consume most of the allowance before application code starts, so measuring only handler execution time produces an unrealistically small estimate.

What Should Happen When Time Expires?

At timeout, the caller should stop waiting, release its local resources, and preserve enough evidence to explain the event. Useful evidence includes the operation name, target, phase, elapsed time, configured deadline, connection state, and a correlation or request identifier.

The client may signal cancellation to the server or close the stream. Cancellation is best effort. The server may already have completed, may ignore the signal, or may be unable to roll back an external side effect. “The caller stopped caring” and “the server stopped working” are separate facts.

This is why timeout handling must represent an unknown business outcome when appropriate. Automatically labeling every timeout as “operation failed” can be more dangerous than the timeout itself.

For an operation with side effects, the most important classification is often not the transport error type but what can be concluded about the result.

Known success means the caller received a valid response that, according to the protocol contract, confirms completion.

Known failure means the caller has evidence that the operation was rejected before taking effect. A validation response can provide this guarantee if the API defines it clearly. A direct connection refusal before any request connection exists also means that request was not executed through that attempted connection.

Unknown outcome means the caller cannot prove either conclusion. A timeout after transmission, a reset during response reading, or a server error returned after partial processing can all fall into this category.

Consider three possible histories behind the same client-side read timeout:

The observed timeout is identical; the business state is not.

APIs should document when an error response guarantees that no side effect occurred. Logs and user-visible messages should avoid claiming more certainty than the protocol provides. A message such as “We could not confirm the transfer” is more accurate than “The transfer failed” when the outcome is unknown.

A partial failure occurs when some components or communication paths fail while the rest of the system continues operating.

This is the defining operational difference between a distributed application and a single all-or-nothing process. A storefront may serve product pages while checkout is broken. Checkout may create an order while failing to load loyalty points. Two replicas may be reachable while a third is isolated.

The whole platform is neither simply “up” nor simply “down.” Its behavior depends on which user action touches which dependency.

Dependency Criticality

The correct response to partial failure depends on the role of the failed dependency.

A product page may safely omit personalized recommendations. Checkout should not claim that payment succeeded when the payment outcome is unknown. A permissions service failure may require denying access, while an analytics failure should not prevent a purchase.

Each dependency should therefore be classified by the business operation:

These are business decisions, not merely networking decisions. Returning stale or partial data is useful only when the API makes that state clear and the domain allows it.

Dependency Graphs Propagate Failure

Suppose service A waits for B, B waits for C, and C becomes slow. B's workers remain occupied while waiting for C. A's workers then remain occupied while waiting for B. The original problem in C consumes resources in every synchronous caller above it.

A partial failure can therefore become a cascade:

The dependency graph determines the blast radius. A small service on the critical path of many APIs can cause more visible damage than a large isolated service.

Fan-out creates another form of exposure. If an aggregator requires results from 50 partitions, one unavailable partition can fail the entire request. If partial results are acceptable, the response contract must say which partitions contributed and whether the result is complete. Silently presenting partial data as complete converts an availability problem into a correctness problem.

A gray failure is a partial failure that looks healthy from some viewpoints and unhealthy from others.

A service process may respond to a lightweight status request while its real operations time out. One availability zone may reach it while another cannot. Reads may succeed while writes fail. Small responses may work while large responses stall. One server instance may fail only for a particular tenant or request shape.

This produces differential observability: different observers have valid but conflicting evidence.

An overall success average can hide the affected group. If 5% of clients have a 100% failure rate, the global dashboard may still show 95% success and obscure a complete outage for those clients.

Diagnosis must therefore segment observations by path and workload: region, zone, instance, client network, protocol version, operation, payload size, and response status. Synthetic status probes are useful, but they do not replace measurements of real operations from real caller locations.

Gray failures are often harder to recover from than clean crashes. A dead endpoint is easy to stop using. A slow endpoint continues accepting work, consumes resources, and may pass simple checks while damaging its callers.

A network partition occurs when the network divides a distributed system into groups that cannot communicate reliably with one another, even though nodes in each group may remain healthy.

A partition can result from a failed link, routing error, firewall rule, broken tunnel, switch failure, overloaded network device, or severe packet loss. It does not require the entire internet or data center to go down.

Both groups can perform local computation and answer local requests. The broken cross-group communication is the failure.

Partitions are not always symmetric. A can sometimes send to B while replies from B cannot return to A. Traffic on one port or protocol may be blocked while another works. Large packets may fail while small packets pass. These selective behaviors can look like latency, omission, or application failure rather than a clean cut.

A Partition Is an Observation Over Time

There is no packet field that announces, “the network is partitioned.” Nodes infer a problem because expected messages do not arrive.

Within a finite observation period, these situations can look identical:

A heartbeat or request timeout can mark a peer as suspected, but silence cannot prove the cause. A short detection window reacts quickly but produces more false suspicions during temporary delay. A long window avoids some false positives but takes longer to react to real failures.

This uncertainty is fundamental in a network with unbounded delay. If a node has not responded yet, another node cannot know whether it will respond one millisecond later or never.

Partitions become dangerous when separated nodes make conflicting decisions.

Suppose a database has a primary in one zone and a replica in another. The zones lose communication. Clients near the original primary can still reach it, so it continues accepting writes. The replica cannot see the primary and is promoted based only on that absence. Clients near the replica now write to the newly promoted primary.

The system has split-brain: two nodes believe they have authority over the same state.

When communication returns, there is no automatically correct way to combine arbitrary conflicting operations. Choosing the latest wall-clock timestamp can lose a valid update and can be wrong when clocks differ. Some domains define safe merge rules; others require preventing concurrent authority in the first place.

A common protection is to require authorization from a majority, or quorum, before a node acts as the current owner. In a three-member group split into two members and one member, only the two-member side can obtain a majority. The isolated member must stop accepting operations that require exclusive authority.

Quorums do not repair the network. They provide a rule that prevents both sides from believing they are authorized at once. Correct implementations also ensure that an old owner cannot continue writing to a shared resource after a new owner is chosen.

Simply declaring a new leader whenever the old one is unreachable is unsafe. Unreachable does not mean dead, and dead does not mean unable to return later with stale authority.

When replicas cannot communicate, a system cannot both maintain one immediately consistent view and successfully process every request on every side.

It has two broad choices for operations that require coordination:

Preserve a single consistent view. A side that cannot prove it has authority rejects or delays the operation. Some clients lose availability, but conflicting updates are prevented.

Continue serving on separated sides. Both sides accept operations without coordination. Availability improves, but state can diverge and the system needs a domain-specific way to reconcile it later.

This is the practical meaning behind the CAP trade-off. CAP applies narrowly to replicated state during a partition:

Partition tolerance is not an optional feature that makes partitions stop occurring. If communication can fail, the design must define what happens. During the failure, it may preserve consistency by refusing some work or preserve availability by accepting potentially divergent work.

Real applications make this choice per operation rather than assigning one label to the entire product. A catalog may serve a slightly stale description during a partition. A uniqueness constraint, inventory decrement, or financial transfer may refuse to proceed without coordination. Reads and writes can also make different trade-offs.

The application must communicate degraded guarantees honestly. A stale read should not be presented as current, and an accepted-but-not-yet-coordinated write should not be presented as globally committed unless the protocol provides that guarantee.

When communication resumes, each side may hold state and in-flight work produced during isolation.

Recovery can involve:

If the system refused conflicting work during the partition, recovery may be mostly a matter of catching replicas up. If both sides accepted updates, recovery must reconcile divergent histories. The correct rule depends on the data: set membership may merge naturally, while two withdrawals from the same limited balance may not.

A node that was isolated can also return with stale configuration, credentials, membership information, or cached data. Restored packet flow proves only that communication works again; it does not prove that every participant now agrees.

Good diagnosis begins with precise observations rather than the phrase “network issue.”

First identify the failed phase. Did resolution fail? Was the client waiting for its pool? Was the connection refused, reset, or established successfully? Did response headers arrive? Was the body complete and valid? Which timer expired?

Then compare both ends using the same operation identifier:

This evidence shows that most of the caller's allowance was spent queueing before the handler began. Increasing application CPU would not address the primary delay.

If the server has no record of the request, inspect earlier stages: name resolution, endpoint selection, connection establishment, proxies, routing, and packet filtering. Absence from application logs does not prove absence from the host; the request may have failed before reaching the handler.

Segment the failure rate by source and destination. A partition or gray failure often appears as one broken pair of zones, one instance, one address family, or one client network inside a healthy global average.

Finally, correlate errors with saturation. Full connection pools, worker queues, file-descriptor limits, CPU pressure, and retransmissions can turn a small dependency problem into widespread timeouts. Packet captures can reveal connection attempts, resets, retransmissions, and one-way traffic, while application traces explain where processing time was spent.

No single observer has the whole truth. Reliable diagnosis combines caller evidence, server evidence, intermediate infrastructure, and the timing relationship between them.

A timeout is not the root cause. It is the caller's decision to stop waiting; overload, packet loss, a partition, or slow processing may be the cause.

A timeout does not prove that an operation failed. The server may have completed the work without delivering the response.

A longer timeout does not make a system more resilient. It may merely retain blocked resources longer and move the failure upstream.

A successful local write to a socket does not prove remote receipt. It normally proves that the local networking stack accepted the bytes.

An HTTP error response is not the same as no response. An explicit response provides evidence from a reachable server or intermediary.

Cancellation does not undo completed work. It communicates that the result is no longer wanted; server-side effects require their own semantics.

Partial failure does not mean the whole system is unavailable. The effect depends on the failed component and the operations that depend on it.

A process can be alive while the service is unusable. Queue exhaustion, selective path failure, and broken dependencies produce gray failure.

Missing heartbeats do not prove that a node crashed. Delay, loss, overload, and partitions produce the same observation.

A partition is not necessarily a clean, symmetric cut. One direction, protocol, path, or message size can fail while another succeeds.

CAP does not say that a system always chooses only two of three properties. It describes the consistency-versus-availability choice forced by a partition.

A healed network does not guarantee healed state. Replicas, owners, caches, and in-flight operations may still disagree.

Distributed failures must be separated into cause, caller observation, and business outcome. Calls can fail during resolution, pool waiting, connection setup, request transmission, remote processing, or response reading. Crash, omission, timing, response, and partition failures may overlap.

Timeouts bound waiting but neither diagnose the cause nor prove remote failure. Phase-specific timeouts plus an end-to-end deadline make service-chain limits explicit. Side-effecting work may end in known success, known failure, or an unknown outcome.

Partial and gray failures leave some paths healthy and can hide inside healthy averages, so dependency criticality determines whether to fail, degrade, or defer. Partitions separate healthy groups and make detection uncertain. Preventing split-brain requires one authority rule; coordinated state must sacrifice some availability for consistency or accept later reconciliation.

Silence proves only that the caller did not observe a result in time—not whether the operation failed, succeeded, or continues.