A server has 10,000 connected clients. At one instant, perhaps twelve of them have sent new data.

Calling blocking read() on the connections one by one cannot work:

The thread reads client 1, waits because client 1 is idle, and clients 2 through 10,000 are never checked.

Checking every connection with non-blocking read() in a tight loop avoids sleeping on one client, but repeatedly tests thousands of idle sockets and burns CPU.

I/O multiplexing solves the waiting problem. The application gives the kernel a set of descriptors and asks:

Put this thread to sleep until at least one descriptor may be ready for an operation.

On Unix-like systems, select() and poll() provide portable forms of this mechanism. Linux provides epoll, which is designed for larger, longer-lived descriptor sets.

Without I/O multiplexing, an application has two unattractive choices:

A multiplexer moves the idle wait into the kernel:

The multiplexer does not normally transfer the application data. It reports that a requested operation may now make progress. The application still calls read(), write(), accept(), or another operation on the reported descriptor.

This produces a common loop:

The loop never exits while the server runs. Every design question in this chapter is about how much work happens per pass and how long the wait step is allowed to last.

The waiting function itself can block efficiently even though the watched sockets use non-blocking mode. There is no contradiction: the thread blocks once on the combined set rather than accidentally blocking on one individual connection.

A descriptor is readable when a read-like operation can return without waiting for more endpoint activity.

For a stream socket, readability can mean:

Readable therefore does not mean “a positive byte count is guaranteed.”

A descriptor is writable when a write-like operation can make some progress without waiting for more local buffer capacity. It does not mean that an entire large payload will fit, nor that a remote peer has received anything.

Readiness is a current observation, not a reservation. Another thread could consume the data before this thread calls read(). The descriptor's state can also change between notification and use.

For that reason, descriptors used in a multiplexed loop are normally non-blocking. The application must still handle EAGAIN as an ordinary race or drain boundary.

All three interfaces can wait indefinitely, wait up to a timeout, or return immediately.

An infinite timeout is useful when the application has nothing to do until I/O changes. A finite timeout lets it regain control for housekeeping or deadlines. A zero timeout performs an immediate readiness check, which becomes polling if repeated continuously.

The wait can also return -1 with errno set to EINTR when a handled signal interrupts it. Correct code usually retries while preserving the application's intended deadline. Restarting a full relative timeout after every signal can accidentally wait longer than the original budget.

A readiness timeout is not a failure on any watched descriptor. It means no requested readiness event became reportable before the wait expired.

select() represents interests with bit sets:

The application uses macros such as:

nfds is one greater than the highest descriptor number in any set. It is not the number of descriptors being watched:

On return, each set contains only the descriptors for which the requested condition is ready. select() therefore modifies the sets supplied by the caller.

A typical loop keeps a master set and copies it before every wait:

The code scans from zero through the highest watched descriptor to discover which bits remain set. A high descriptor number can therefore increase scanning even when only a few descriptors are registered.

Timeout state also needs care. Linux can update the supplied timeval to reflect unslept time, while other systems differ. Portable loops rebuild both the working descriptor sets and the timeout value before each call.

The FD_SETSIZE limit

fd_set has a fixed representable size in common implementations. On many systems, including typical glibc builds, FD_SETSIZE is 1024.

Using FD_SET() with a descriptor outside the set's representable range is unsafe. Increasing the process's open-file limit does not automatically enlarge an already compiled fd_set.

This numeric-descriptor limit and the repeated set scanning make select() a poor fit for high-connection-count Linux servers. It remains useful for small portable programs and for understanding the basic model.

poll() replaces bit sets with an array of structures:

Each entry contains:

For example:

After poll() returns, the application examines revents for every array entry:

Common event bits include:

Errors and hangups can be reported even when they were not explicitly requested. A hangup can appear together with readable data, so code should consume any remaining input before treating the stream as finished.

Unlike select(), poll() does not use a fixed-size bit set and does not care whether a watched descriptor's numeric value is large. The process's resource limits and available memory still bound how many descriptors can exist.

The array must still cross the user/kernel boundary for each call, and the kernel examines its entries. The application also scans the array after return to find nonzero revents. For a large, mostly idle set, that repeated work remains proportional to the number of watched entries.

Linux epoll separates three operations:

Create an instance:

Register a descriptor:

Wait for ready events:

epoll_wait() returns up to 64 ready event records in this example. The application processes only those returned entries:

The epoll instance itself is represented by a file descriptor. The kernel keeps its registered interest list between waits and maintains a ready list for entries with reportable events:

The interest list stays in the kernel between calls, and only the short ready list crosses back. That is the difference that lets epoll scale past select.

The application does not resend all six registrations for every wait. It calls epoll_ctl() only when the watched set or an entry's interest changes.

By default, epoll uses level-triggered reporting: a condition can remain reportable while it remains true. More specialized notification flags change when events are generated, but persistent registration and ready-only return are the core ideas needed here.

Assume 10,000 sockets are registered and only twelve are ready.

With select():

With poll():

With epoll:

The practical comparison is:

This is why the statement “epoll is O(1)” is too vague to be useful. Registration changes, kernel bookkeeping, wakeups, and processing each returned event all cost work. If 10,000 descriptors are ready, the application must handle 10,000 events.

The advantage is strongest when a large stable set contains relatively few ready descriptors per wakeup. With ten descriptors, the simpler interfaces may perform perfectly well. Frequent registration changes can also make epoll_ctl() costs relevant.

select(), poll(), and epoll primarily report readiness.

For example:

epoll_wait() reports the socket readable, the application calls recv(), and recv() returns bytes, 0, or an error.

The event is not a completed recv() operation and does not contain the received application bytes. It says that attempting the operation is meaningful now.

This also explains why a non-blocking operation can still return EAGAIN after a readiness notification. Readiness may be consumed by another thread, or the application may already have drained the endpoint before processing another queued notification.

I/O multiplexing is most useful for endpoints whose readiness changes over time, such as sockets, pipes, terminals, and some device descriptors.

Ordinary regular files do not behave like network sockets. On Linux, select() and poll() generally consider a regular file ready because a read or write does not wait for a peer to produce data or buffer capacity in the same sense.

Registering an ordinary regular file with Linux epoll commonly fails with EPERM. epoll is not a general notification mechanism for physical disk completion.

A loop that repeatedly receives “regular file readable” can still block elsewhere in filesystem or storage work. Readiness describes the operation's endpoint contract, not every internal source of latency.

The following Linux server listens on 127.0.0.1:8080, accepts non-blocking connections, and reports how many bytes it receives. It deliberately does not implement an application protocol; the focus is descriptor registration and readiness handling.

Compile it on Linux:

In another terminal, connect with:

Anything typed into nc becomes readable on the accepted socket. Closing nc produces end-of-stream or a peer-shutdown event, and the server removes the client from epoll before closing it.

Several details are essential:

The loop handles client events before accepting new descriptors. This prevents a descriptor number closed during the current event batch from being immediately reused by accept4() while an older event with the same numeric value is still unprocessed.

Diagnostic printf() calls are convenient for learning but can themselves block if standard output is slow. Production event loops send diagnostics through a bounded, non-blocking logging design or keep synchronous output away from the I/O loop.

Observe the control flow

Run the server under strace:

The trace shows a persistent pattern:

Registrations change when connections open or close. The full descriptor set is not supplied again on every epoll_wait().

Blocking after readiness

Readiness can become stale. If the socket remains in blocking mode, a supposedly safe read() can still park the entire loop. Use non-blocking descriptors and handle EAGAIN.

Watching writable sockets all the time

A healthy socket is often writable. Permanently registering POLLOUT or EPOLLOUT can make the wait return continuously even when the application has nothing to send.

Watch for writability only while output remains buffered, then remove that interest after the pending data is accepted.

Closing immediately on hangup

POLLHUP or EPOLLHUP can accompany readable bytes. Process the readable state before discarding the connection so that already queued input is not lost.

Treating readiness as a completed operation

An event means an operation may make progress. The subsequent read(), write(), or accept() produces the actual result.

Reusing select() sets without rebuilding them

select() overwrites its descriptor sets with the ready subset. Reusing that subset directly silently stops watching descriptors that were not ready during the last call.

Passing the wrong nfds to select()

The first argument is the highest watched descriptor plus one, not the number of bits currently set.

Ignoring error event bits

Readiness APIs can report errors and hangups separately from ordinary read/write interests. Looking only for POLLIN or EPOLLIN can leave broken descriptors registered and cause repeated wakeups.

Assuming epoll is always faster

For small sets, scanning cost is tiny and simpler APIs may be entirely adequate. epoll is most valuable for large, stable, mostly idle sets.

Assuming one event means one byte or one request

Readiness coalesces state. One notification may correspond to many queued bytes or several pending connections. The application must continue until the endpoint's current work is consumed.

I/O multiplexing lets one thread sleep until at least one of many descriptors may support useful I/O. The multiplexer reports readiness; the application still performs the read(), write(), or accept() that produces the result.

select() uses fixed-size descriptor bit sets, modifies them on return, and requires scanning through the highest watched descriptor. poll() removes the bit-set limit and uses an array, but the complete array is still supplied and scanned on every call.

Linux epoll maintains a persistent interest list and returns entries from a ready list. This avoids repeatedly copying and scanning thousands of idle registrations, although processing ready events and changing registrations still costs work.

Readiness is not a reservation. Multiplexed descriptors should be non-blocking, and code must handle partial progress, EOF, errors, hangups, and EAGAIN. Writable interest should be enabled only when output is pending to avoid a busy loop.

The central mental model is:

select, poll, and epoll answer which descriptors may make progress now; they do not perform the I/O for the application.