A client reports that an API is “slow to connect.” On the server, CPU utilization is moderate, the process is running, and the listening port appears open. Those facts do not explain whether the client is waiting for name resolution, routing, TCP establishment, the server application, or data transfer.

Network investigations become tractable when the end-to-end operation is divided into observable phases:

Each phase has different evidence. A DNS delay cannot be diagnosed from a socket's send queue. An established TCP connection does not prove that the application is responding. A retransmission counter does not identify which client or path was affected.

The objective is to connect a precise client-visible symptom to a specific socket, network namespace, queue, counter, or packet sequence.

Begin with the operation that failed and the phase in which it failed.

Name-resolution failures occur before a client has selected a destination address. The application may report an unknown host, a resolver timeout, or inconsistent addresses across attempts.

Connection-establishment failures occur while creating a transport connection. Common symptoms include immediate refusal, a connection timeout, or intermittent success across destination addresses.

Established-connection failures occur after setup succeeds. Requests may wait for a response, transfers may make little progress, or connections may reset unexpectedly.

Server-admission failures occur when a service is nominally listening but cannot accept new work quickly enough. Existing connections may remain healthy while new clients experience delays.

Local binding failures prevent a server from creating its endpoint. The address may not exist in the current namespace, another socket may conflict with the requested address and port, or the process may lack permission.

Write the symptom in terms of a direction and endpoint:

This is more useful than “the network was down.” It identifies the source, destination, port, transport phase, and time window.

Network state is scoped. A container can have different interfaces, routes, firewall rules, socket tables, and DNS configuration from the host.

Compare the process and shell network namespaces:

If the links identify different namespace objects, commands run in the host shell may not describe what PID 2471 sees. With appropriate permission, run a diagnostic in the process's network namespace:

The same principle applies to DNS and endpoint tests. A lookup from an engineer's laptop does not establish what a container resolved. A connection from the host namespace may use a different source address, route, firewall path, or proxy than the application.

Also preserve the address family. A hostname may resolve to both IPv4 and IPv6, and only one path may be failing. Record the actual remote address chosen by the client rather than testing whichever address is convenient.

An application transaction crosses several boundaries:

Measure these phases separately when the client supports it. For a known, safe HTTP health endpoint:

This command performs a real request. Use it only on an endpoint known to be safe and side-effect free.

The time values are cumulative from the start of the operation. time_connect includes earlier lookup time, and time_starttransfer includes lookup, connection setup, any secure handshake, request transmission, and time spent waiting for the first response byte. Subtract adjacent timestamps to estimate individual phases.

Interpret the result as a client observation:

This does not prove where inside a phase the delay occurred. It tells you which layer to investigate next.

On a typical Linux system, applications resolve names through the Name Service Switch rather than by querying DNS directly. Inspect the result through that path:

The result can reflect /etc/hosts, DNS, and other sources configured in /etc/nsswitch.conf. Run it in the same container or network context as the application.

Inspect the resolver configuration without assuming /etc/resolv.conf is a regular, manually maintained file:

On a system using systemd-resolved, this can provide additional evidence:

Tools such as dig are valuable for testing a particular DNS server, but they do not necessarily reproduce the application's complete NSS behavior, local host mappings, caching layer, search rules, or address selection. Use them to answer a specific DNS question, not as automatic proof that application resolution is healthy.

When a name returns several addresses, record all of them and the one selected by the failed attempt. Intermittent failures often arise because only one address, address family, or backend path is unhealthy.

Inspect the addresses and operational state of interfaces:

Then ask the kernel how it would route to the actual destination address:

The output normally includes the selected route, next hop when applicable, outgoing device, and source address. ip route get performs a route lookup as the kernel sees it but does not transmit a packet.

This is more precise than inspecting only the default route. Policy routing, source-address selection, virtual routing tables, and more specific prefixes can direct one destination differently from another. If the real traffic uses a particular source address, mark, protocol, or interface, a basic lookup may need matching qualifiers.

For a destination on a directly connected link, inspect neighbor resolution:

A transient neighbor state is not automatically a failure. Repeated unresolved or failed neighbor entries during the symptom can explain why packets never leave the local link.

ping is only a limited reachability test. ICMP echo can be blocked while the application port works, and a successful ping does not test DNS, TCP admission, secure negotiation, or application processing. Treat its result as evidence about one protocol path.

On the server, locate the expected TCP listener:

The options select listening TCP sockets, keep addresses and ports numeric, and request process ownership. A simplified result looks like:

Check four details:

The address determines where the service accepts traffic. A listener on 127.0.0.1:8443 is not reachable through the host's external address. An IPv6 wildcard listener and an IPv4 wildcard listener may behave differently depending on socket options and system configuration.

The port must match what the client and any load balancer target.

The owning process connects the socket to the service instance under investigation. Process details may be hidden without sufficient permission.

The namespace determines which interfaces, routes, and port space contain the listener. A host-level proxy may listen on one socket and forward to a container listener in another namespace.

More than one process can legitimately own listeners for the same endpoint through mechanisms such as socket activation or SO_REUSEPORT. Do not assume the first displayed PID is the complete serving topology.

For UDP and local Unix sockets:

UDP has bound sockets but no TCP-style LISTEN or ESTAB state. Unix domain sockets do not use IP routing or DNS, so focus on their pathname or abstract name, permissions, queues, and owner.

A compact system summary is useful before dumping thousands of sockets:

It reports counts by protocol and broad TCP state. The values are a snapshot, not rates. A large count on a busy long-running server may be normal.

Filter to the service or dependency of interest:

For an outbound database connection:

Filtering improves both performance and interpretation. It prevents unrelated traffic, such as monitoring agents, service discovery, administration, and other applications, from contaminating the conclusion.

TCP state describes protocol progress, not application health.

SYN-SENT means the local endpoint has initiated connection establishment and is waiting for the required response. A sustained rise among failed clients suggests that establishment is not completing.

SYN-RECV means a server has received an initial request and is waiting for the final establishment step. Some entries are normal during active connection creation. A large sustained population during failures suggests incomplete handshakes, loss, overload, or unwanted traffic.

ESTAB means transport establishment completed. It does not prove that the application accepted useful work, read a request, or produced a response.

CLOSE-WAIT means the peer has closed its sending direction and the local application has not yet closed its side. Individual entries are normal while cleanup occurs. A continuously growing population tied to one process suggests that the application is not closing promptly.

FIN-WAIT-2 means the local side's close was acknowledged, but it is still waiting for the peer's close. Persistent growth can indicate a peer or closure-path problem.

TIME-WAIT is kernel-maintained state after the endpoint that actively closed a connection. It protects new connections from delayed packets belonging to an older connection. A high count often reflects high connection churn and is not by itself a socket leak.

Sample states over time. A thousand TIME-WAIT entries that drain at the expected rate tell a different story from a monotonically growing CLOSE-WAIT population.

ss shows Recv-Q and Send-Q, but their meanings depend on socket state.

For an established TCP connection:

A persistently growing Recv-Q suggests that the local application is not reading as fast as data arrives. It may be busy, blocked, paused, or deliberately applying application-level backpressure.

A persistently large Send-Q means the local side has data outstanding. Possible causes include packet loss, congestion, a constrained path, or a remote application that is not reading and has reduced its advertised receive window. The queue alone cannot distinguish those causes.

For a listening TCP socket, the columns have different units:

A listener whose Recv-Q repeatedly approaches Send-Q is under admission pressure. Confirm the impact with connection latency and listen-overflow counters rather than declaring failure from one snapshot.

Zero queues do not prove a service is healthy. A request can wait in an application queue after its bytes have been read, or a server can simply have received no request.

Request detailed TCP and socket-memory information for a narrow set of connections:

-i requests internal TCP information. Depending on kernel and iproute2 versions, fields can include:

-m adds socket-memory accounting, including receive and send allocations, configured buffer limits, queued write memory, backlog memory, and socket drops.

Read these values in context. TCP RTT is not application request latency. It measures transport acknowledgments, not how long the remote service spends processing a query. A high retransmission count accumulated over a long-lived connection is not proof that retransmissions are occurring now. Compare repeated samples or a healthy connection with similar age and traffic.

Per-socket evidence is particularly useful for asymmetry. If only connections to one destination have rising retransmissions and send queues while other destinations on the same host are healthy, a host-wide application problem becomes less likely.

Interface counters are cumulative. Inspect them on the selected route's device:

Take two snapshots over a known interval and calculate the change. Relevant fields include received and transmitted packets and bytes, errors, drops, overruns, carrier errors, and collisions. The exact meaning and quality of lower-level counters depend on the driver and device.

Driver-specific evidence may be available through:

Counter names vary by driver. Virtual interfaces and cloud devices may not expose physical-link health. A counter that never changes may be unsupported rather than proof that the event cannot occur.

Kernel protocol counters can be read with nstat:

-a requests absolute values rather than increments maintained in nstat's history, and -z includes zero-valued counters. Capture two absolute snapshots so the incident calculation does not depend on who last ran the tool.

Interpret the deltas:

Host counters aggregate many flows unless a namespace narrows the scope. They can establish that a class of event occurred during the window, but not which request experienced it.

Client-visible errors imply different packet-level outcomes.

An immediate connection refused usually means the destination or an intermediary responded promptly that the connection was not accepted. Common explanations include no listener at that address and port, a listener in a different namespace or address family, or a policy configured to reject.

A connection timeout means establishment did not complete before the client's deadline. Packets may have been dropped, routed incorrectly, lost on the return path, blocked by policy, or sent to an unavailable endpoint. The timeout alone cannot locate the loss.

A connection reset means an endpoint or intermediary aborted an existing or attempted connection with a reset. The reset's source and timing matter: immediately after establishment, during protocol exchange, and after an idle period suggest different mechanisms.

An application read timeout after an established connection does not imply the network dropped the request. The server may have received the request but failed to produce a response. Compare socket queues, packet timing, and server logs before assigning the cause to the network.

Socket state tells you what the local kernel believes. A packet capture shows what crossed a selected observation point.

Capture a small, filtered sample:

This keeps addresses and ports numeric, adds full timestamps, stops after 100 packets, and limits capture to the endpoint under investigation. The any interface is convenient for discovering the path, but traffic can appear at multiple virtual layers. Once the interface is known, capturing on that specific device gives a clearer boundary.

A handshake capture can distinguish several patterns:

One capture cannot identify where an absent packet disappeared. Capture at both endpoints, or at two controlled boundaries, and align clocks when the location matters.

Packet capture requires care. It can expose payloads, credentials, internal addresses, and customer data. Apply strict filters and duration limits, store output securely, and obtain required authorization. Checksum offload can also make outbound packets appear to have invalid checksums in a host capture even though the network device fixes them before transmission.

UDP does not establish a connection or retransmit lost datagrams on behalf of the application. A successful local send only means the local kernel accepted the datagram for transmission.

Inspect UDP endpoints and queues:

A growing receive queue indicates that the application is not draining datagrams quickly enough. Confirm loss with changes in socket drops and UDP error counters. Because there is no TCP-style backpressure, a receiver can lose datagrams while the sender continues successfully.

Unix domain sockets use the socket API without IP networking:

For a pathname socket, confirm both the filesystem entry and a live kernel listener. A stale path can remain after a process exits. Abstract socket names, commonly shown with an @ prefix, have no filesystem entry. Routes, DNS, NIC errors, and TCP retransmissions are irrelevant to these local endpoints.

An API's existing connections remain healthy, but connection p99 rises sharply during brief traffic bursts. Some clients time out. Server CPU stays below 50%, and the application reports no increase in request-processing time.

Client timing shows that DNS is fast but the connection phase is slow. Captures from an affected client show repeated SYN attempts during each burst.

On the server, the listener exists:

During a burst, its output repeatedly reaches:

For this listening socket, 128 completed connections are waiting for accept(), equal to the displayed accept-backlog limit.

Two snapshots of absolute kernel counters taken ten seconds apart show:

The listener queue fills at the same time as client establishment latency rises, and the kernel records new listen overflows and drops. This is much stronger evidence than the full queue snapshot alone.

Configuration comparison reveals that the latest release stopped supplying the service's explicit backlog of 1,024. The framework defaulted to 128. The server can drain the burst shortly afterward, which explains why CPU remains moderate and existing connections are unaffected, but the smaller queue cannot absorb the arrival spike.

Restoring the configured backlog changes the listener's displayed Send-Q to 1,024. Under the same burst, Recv-Q briefly peaks around 430, the overflow counters no longer increase, and client connection latency remains within its normal range.

The root cause is:

A deployment regression reduced the API listener's accept backlog from 1,024 to 128. Short connection bursts filled the smaller queue, causing listen drops and client retransmission delays before the application could accept the connections.

The durable evidence connects client timing, packet behavior, server queue state, kernel counter deltas, and a configuration change.

Use this sequence to keep a network investigation focused:

Stop when the evidence identifies both the failing phase and the mechanism that prevents progress.

Network diagnosis begins by identifying the phase that failed: name resolution, routing, connection establishment, application response, data transfer, or closure. Run every check from the relevant namespace and preserve the actual address family, source, destination, and time window.

Use ss to connect sockets to processes, states, queues, and per-connection TCP evidence. Use ip for local addresses and route selection, and measure interface and protocol counters as deltas rather than isolated cumulative totals. A bounded packet capture can then clarify the packet sequence when higher-level evidence remains ambiguous.

The final diagnosis should explain both ends of the symptom: what the client waited for and which server, socket, route, queue, or packet event prevented progress.