A web server writes access logs while a separate process compresses them. The producer emits a sequence of bytes, the consumer reads them in order, and neither process needs random access to old data. A pipe matches this flow.

The shell uses the same mechanism for a command such as:

Each command runs in a separate process. The shell creates kernel-managed channels, connects one command's standard output to the next command's standard input, and closes its own copies of those channels. Data then moves through the pipeline as each command reads and writes.

Pipes look simple because read() and write() are familiar. Their correctness depends on details outside those calls: which process still owns each descriptor, when the kernel reports end-of-file, how a bounded buffer slows a producer, and whether several writers can interleave their data.

The pipe() system call creates one kernel pipe object and returns two file descriptors:

By convention:

Bytes written to the write end enter a bounded kernel buffer. A read from the read end removes bytes from that buffer:

The pipe preserves byte order. If a single writer successfully writes ABC and then DEF, a reader receives those bytes in the order ABCDEF.

The kernel does not preserve the two write calls as two records for the reader. One read may return ABCDEF, two reads may return AB and CDEF, or several smaller reads may divide the stream again. Pipes carry bytes rather than messages.

A pipe provides ordered delivery and kernel-managed buffering. The application still defines where one logical record ends and the next begins.

A text protocol can use newline delimiters. A binary protocol can place a fixed-size length field before each payload. The reader must accumulate bytes until one complete record is available.

Pipes do not support seeking. Calling lseek() on a pipe fails with ESPIPE because consumed bytes are removed rather than retained as randomly addressable data.

An anonymous pipe has no pathname or global name. A process normally creates it before fork(), allowing the child to inherit both descriptors.

After fork(), the parent and child each have descriptor table entries that refer to the same kernel pipe:

The intended direction determines which copies remain open. For parent-to-child data:

The numeric descriptor values belong to each process. The important connection is the referenced kernel object, not whether both processes happen to use descriptor 3 or 4.

A POSIX pipe is a one-way channel. Full-duplex communication requires two pipes, one for each direction. That arrangement needs a protocol that prevents both processes from filling their outgoing pipe while neither reads its incoming pipe.

For a two-command pipeline, the shell creates the pipe before it creates either command:

The producer must see the pipe's write end as standard output, descriptor 1. The consumer must see the read end as standard input, descriptor 0. dup2() creates those mappings.

Every process then closes the original descriptors it no longer needs. The shell closes both ends because it should neither produce nor consume pipeline data. It waits for the children only after closing them.

A three-command pipeline contains two pipe objects. The middle command uses the first pipe as standard input and the second pipe as standard output. The shell creates the complete descriptor topology before waiting for any command, which allows all stages to make progress concurrently.

The following program implements printf '%s\n' alpha beta | wc -l. It uses the portable pipe() interface and marks the original pipe descriptors close-on-exec.

Compile and run it:

wc prints a line count of 2.

FD_CLOEXEC prevents an original pipe descriptor from leaking into a newly executed program. dup2() creates the standard-input or standard-output descriptor that the new program needs. That descriptor remains open across exec().

Linux also provides:

pipe2() creates both descriptors with close-on-exec enabled in one operation. This avoids the interval between pipe() and fcntl() in which another thread could create a child process that inherits the descriptors. The portable version keeps the example usable on other POSIX systems, while multithreaded Linux programs generally prefer pipe2().

The kernel tracks every open reference to each end of a pipe. A duplicate created by fork() or dup() counts, as does a descriptor inherited across exec().

A reader receives end-of-file only after both conditions hold:

This explains a common pipeline hang. The producer finishes and closes its write descriptor, but the parent still owns another write descriptor. The consumer drains the buffered data and calls read() again. The kernel sees an open writer, so it waits for more data instead of returning end-of-file.

Closing the parent's copy changes the outcome:

The write side has a corresponding rule. If no read descriptors remain, the kernel sends the writer SIGPIPE. The default action terminates the process. If the process catches or ignores that signal, write() fails with EPIPE.

SIGPIPE lets a pipeline stop upstream work when downstream processing has ended. For example, head may read ten lines and exit while its producer still has more output. The producer then learns that no consumer remains.

Programs that ignore SIGPIPE must check for EPIPE. Repeatedly retrying the same write cannot restore a reader.

read(descriptor, buffer, capacity) returns the number of bytes placed in buffer. A positive result can be smaller than capacity even when more bytes may arrive later.

Correct stream processing uses a loop:

The consumer passes exactly count bytes onward. It cannot treat the buffer as a null-terminated string unless it reserves space and inserts the terminator itself.

write() can also return after accepting fewer bytes than requested, especially for large writes, interrupted calls, or nonblocking descriptors. A writer that must deliver the complete buffer advances by the returned count and continues until all bytes have been accepted or an unrecoverable error occurs.

That loop does not make a large logical message atomic relative to other writers. If several processes write to one pipe, their separate retry loops can alternate. Message framing and atomic-write limits still matter.

A pipe buffer has finite capacity. The exact capacity is an implementation detail and can change with the operating system, kernel configuration, and resource limits.

When a blocking writer finds insufficient space, the kernel can put it to sleep. A reader removes bytes, which frees space and lets a writer run again. When the buffer is empty but at least one writer remains, a blocking reader sleeps until data arrives or the last writer closes.

This gives a pipeline natural backpressure:

The producer's throughput eventually matches the consumer's sustained throughput. Memory use stays bounded instead of growing with the difference between their rates.

Blocking behavior also creates deadlock risks in two-way protocols. Two processes can each fill one pipe while waiting to write more, even though neither has started reading from the other pipe. Protocols should define when each side reads and writes, and large bidirectional exchanges often need concurrent reading.

On Linux, fcntl(descriptor, F_GETPIPE_SZ) reports a pipe's capacity in bytes. F_SETPIPE_SZ can request another size, subject to permissions and system limits. Code should remain correct at any allowed capacity rather than depending on a particular default.

Setting O_NONBLOCK changes full and empty cases into immediate errors such as EAGAIN. That mode is useful when one thread manages several I/O sources, but it does not change the pipe's byte-stream or capacity semantics.

Pipe capacity answers:

How many bytes can the kernel buffer before writers must wait or receive EAGAIN?

PIPE_BUF answers:

How large can one write be while retaining the atomic-write guarantee relative to other writers?

POSIX guarantees that writes of at most PIPE_BUF bytes are not interleaved with competing writes to the same pipe. If Writer A submits one 100-byte record and Writer B submits another 100-byte record, each through one write() call, a reader receives one complete record followed by the other when both sizes fit within PIPE_BUF.

The order between writers is unspecified. Atomicity prevents their bytes from mixing; it does not choose which writer goes first.

A write larger than PIPE_BUF may be split and interleaved:

This trace illustrates interleaving rather than specific chunk sizes.

Applications can query the limit for a pipe descriptor:

On Linux, PIPE_BUF is commonly 4096 bytes. Pipe capacity is usually much larger. Treating the total buffer capacity as the atomic-write limit can corrupt records when several writers share one pipe.

Even one atomic write can be split across several reads. Atomic write semantics protect writers from interleaving; they do not create read-side message boundaries.

Several processes may hold the same pipe endpoints. The result remains one byte stream.

With multiple writers, all bytes enter the same stream. Writes within the PIPE_BUF rule remain intact, while larger writes can interleave. A delimiter or length prefix must still identify records.

With multiple readers, bytes are consumed by whichever reader receives them. Reading is destructive: one byte removed by Reader A is no longer available to Reader B. A pipe therefore distributes work among readers rather than broadcasting every byte to all of them.

The kernel does not promise an application-level fairness policy among competing readers or writers. A design that requires fair assignment, per-client queues, or broadcast delivery needs those properties implemented above the pipe or provided by another mechanism.

A FIFO, also called a named pipe, exposes pipe-style byte-stream semantics through a filesystem entry. mkfifo() creates that special file:

The shell provides the mkfifo command:

The directory entry persists until it is unlinked, but payload bytes are not stored in the directory or retained like file contents. Once processes open both sides, the kernel maintains a pipe buffer. Bytes disappear when readers consume them, and buffered bytes vanish after the final endpoints close.

The name lets independently started processes connect:

After open() succeeds, processes use read(), write(), and close() as they would with an anonymous pipe.

A blocking open of a FIFO normally waits for the opposite side:

This prevents a writer from sending data before any reader can receive it, but it can make startup order look like a hang. Two terminals make the handshake visible.

In the first terminal, create and read the FIFO:

The shell blocks while opening the FIFO for the loop because no writer exists yet.

In a second terminal, open it by writing:

The reader prints both records. When printf exits, its write descriptor closes. The reader drains the buffer, receives end-of-file, and exits its loop.

Remove the unused name afterward:

Nonblocking open changes the handshake. On Linux, opening the read end with O_NONBLOCK can succeed without a writer, while opening the write end with O_NONBLOCK fails with ENXIO when no reader exists.

Linux also permits opening a FIFO with O_RDWR, even when no peer has opened it. POSIX leaves that case unspecified. Self-opening can prevent end-of-file because the process keeps a write reference alive, so portable protocols should use explicit read-only and write-only endpoints.

The FIFO name and the open channel have separate lifetimes. Unlinking the filesystem entry prevents new opens through that name, while processes that already hold descriptors can continue communicating until they close them.

File ownership, mode bits, and directory permissions control who can open a FIFO. A service should create named pipes inside a directory it controls rather than placing a predictable name in a shared writable directory. mkfifo() fails with EEXIST if an entry already occupies the path; code should not assume that an existing entry is the expected FIFO.

Permissions restrict access, but they do not identify the peer after opening. Any process allowed to open the endpoint can participate. Multiple readers compete for bytes, and multiple writers contribute to the same stream.

A writer may block forever if the expected reader never starts. A reader may wait forever if a writer opens the FIFO but never produces data. Supervisors, timeouts, or nonblocking operation provide the surrounding liveness policy. The FIFO itself reports endpoint closure and I/O errors, but it does not know whether a peer has completed useful work.

Each pipe transfer normally copies data from the producer's user-space buffer into kernel-managed pages and later into the consumer's user-space buffer. It also requires read() and write() system calls. Batching small records reduces calls and wakeups, but excessive batching raises latency.

Pipes are effective for local streams because the kernel supplies ordering, bounded buffering, blocking, wakeups, and endpoint-lifetime tracking. Those services account for some of their cost.

On Linux, strace can expose descriptor setup and data flow:

The trace shows the pipe creation, each child mapping an endpoint onto a standard descriptor, unused-end closures, writes from printf, and reads by wc.

Open descriptors are also visible through /proc:

Pipe descriptors appear as links such as pipe:[123456]. Matching identifiers show which descriptors refer to the same kernel pipe object. When a reader waits indefinitely for end-of-file, inspecting every process in the pipeline often reveals an unexpected write descriptor that remains open.

Anonymous pipes fit process hierarchies in which a parent can create the channel before starting children. Shell pipelines, worker startup, and capturing a child process's output use this pattern.

FIFOs fit local processes that start independently but need a named, one-way byte stream. They add filesystem discovery and permissions while retaining pipe behavior.

Both mechanisms work best when communication is sequential and the protocol can tolerate a stream interface. Request-response communication requires two channels and careful shutdown rules. Broadcast delivery, preserved message records, random access, and communication across hosts require different semantics.

A pipe is an ordered, bounded byte stream held by the kernel. pipe() returns a read descriptor and a write descriptor, which processes usually share through fork() and connect to standard input or output with dup2().

Descriptor ownership determines shutdown behavior. A reader receives end-of-file only after the buffer is empty and every write reference has closed. A writer with no readers receives SIGPIPE or an EPIPE error. Partial I/O, stream framing, bounded capacity, and the PIPE_BUF atomic-write limit all shape a correct protocol.

A FIFO gives pipe semantics a filesystem name, allowing independently started local processes to connect. Its directory entry persists, while its payload remains transient and follows the same read, write, blocking, and endpoint-lifetime rules as an anonymous pipe.