A server receives a 64 KB network response. The bytes arrive through a network interface, land in main memory, and become available to the kernel.

If the CPU had to read every byte from a device register, it would spend thousands of instructions acting as a copy engine. If it had to check the device continuously, it would waste CPU time whenever no data was ready.

Modern I/O hardware divides the work differently:

This division lets the CPU initiate an I/O operation, execute unrelated instructions while the device works, and return to the request when meaningful progress has occurred.

A device driver is kernel software. It understands the operating system's device model and the programming interface exposed by a particular controller.

A device controller is hardware. It provides the CPU-facing mechanism used to submit commands, transfer data, report status, and signal completion. The controller may be integrated into the device, placed on an expansion card, or built into the system chipset.

For example:

The boundary looks like this:

The driver is the last piece of software in the path. Everything below the labelled edge is hardware that the kernel can only address through registers and queues.

The operating system calls a driver operation. The driver converts that request into the controller's command format. This separation lets the rest of the kernel use a stable abstraction while drivers handle hardware-specific details.

A controller exposes a programming interface. Although the details vary, the interface normally provides some combination of:

The CPU must have a way to access this interface. A common method is memory-mapped I/O, abbreviated MMIO. The controller's registers appear at addresses reserved for device communication. When the driver loads from or stores to those addresses, the operation goes to the controller rather than to ordinary RAM.

Some architectures also support a separate I/O-address space accessed by special instructions. The programming model differs, but the purpose is the same: provide controlled access to controller registers.

Device registers are not ordinary variables. Reads and writes can have side effects, and the order of operations may be part of the hardware protocol. A driver therefore uses kernel and architecture-specific access functions that preserve the required access width and ordering.

Registers and queues

Simple hardware may expose a data register for the byte being transferred, a status register containing ready or error bits, and a control register that selects the operation.

High-throughput devices usually use queues in main memory. The driver fills a command descriptor that describes an operation and places it in a submission queue. It then writes a controller register—often described as ringing a doorbell—to announce that new work is available.

A descriptor may contain:

The controller reads queued descriptors, performs the operations, and records completion information. Queues allow many requests to be in flight at once instead of requiring the CPU to wait after every command.

The simplest transfer model is programmed I/O, often shortened to PIO. The CPU executes instructions that read data from or write data to controller registers.

Suppose a simple output device accepts one byte through a data register:

PIO is straightforward and can be appropriate for small control operations or simple devices. Its limitation is CPU cost. Moving a 64 KB payload one byte or word at a time consumes CPU instructions for the entire transfer.

The polling loop offers low response latency because the CPU notices completion quickly, but it occupies a CPU that could otherwise run application or kernel work.

The first major improvement is to let the device notify the CPU instead of making the CPU ask continuously.

An interrupt is an asynchronous notification that causes a CPU to enter a kernel handler. For device I/O, it commonly means that a controller has completed work, received data, detected an error, or needs more resources.

The lifecycle is:

The interrupt does not contain the complete I/O result. It directs the kernel to the relevant handler, which then consults device status or a completion queue to discover what happened.

From the device to a CPU

An interrupt controller sits between device interrupt sources and CPU cores. It identifies interrupt sources, applies routing and priority rules, and delivers a notification to an eligible CPU.

Traditional devices could assert a physical interrupt line. Several devices might share a line, so each relevant driver had to check whether its hardware caused the interrupt.

Modern PCI Express devices commonly use message-signaled interrupts. Instead of asserting a dedicated wire, the device performs a specially formatted write that the platform interprets as an interrupt. MSI-X extends this mechanism with multiple interrupt vectors.

Multiple vectors are valuable for high-throughput devices. A NIC or NVMe controller can maintain several independent queues and associate them with different interrupt vectors. The operating system can then route queue completions across CPU cores rather than sending all device work to one core.

The interrupt handler

Interrupt handling delays whatever the selected CPU was doing, so the immediate handler should perform only time-sensitive work. It typically:

The more expensive work—processing packets, completing many requests, allocating replacement buffers, or waking tasks—can then run outside the most restrictive interrupt context.

This split keeps interrupt latency bounded. A handler that spends too long on one device delays other interrupts and ordinary kernel work on that CPU.

Completion is not immediate application execution

A device interrupt means only that a controller reported an event. Several steps can remain:

The interrupted CPU also does not have to be running the application that issued the request. Any CPU to which the interrupt is routed can begin the completion work.

Interrupts remove the need to wait continuously, but they do not remove the cost of having the CPU copy every byte through a controller register.

Direct memory access allows a DMA-capable controller to transfer data directly between a device and main memory after the CPU has configured the operation.

For a device-to-memory transfer, the division of responsibility is:

The CPU is free during the transfer. It never touches the data moving into memory, which is what separates DMA from the CPU copying bytes register by register.

“Direct” means that the bulk transfer does not pass through CPU load-and-store instructions one unit at a time. It does not mean that the operating system or CPU is uninvolved.

The CPU still:

DMA changes who performs the repetitive data movement, not who controls the operation.

Transfer directions

DMA can move data in either direction:

The direction matters because it determines which component produces new data and what cache or synchronization work is required before the other component reads it.

Consider a network interface receiving a packet. The driver cannot wait for a packet and then search for memory. It prepares receive capacity in advance.

1. The driver prepares buffers

The driver obtains memory buffers that can hold incoming packets. It maps those buffers so the controller has device-visible addresses for them.

2. The driver fills a receive ring

The driver writes descriptors into a circular queue called a receive ring. Each descriptor points to an available buffer and records its capacity.

The driver notifies the controller that new receive descriptors are available.

3. A packet arrives

The NIC receives bytes from the network and selects an available descriptor. Its DMA engine writes the packet into the corresponding RAM buffer. The CPU does not copy each packet byte from a NIC register.

4. The controller completes the descriptor

The controller updates descriptor state with information such as the received length and hardware status. Ownership of that entry changes from the controller back to the driver.

5. The controller notifies the kernel

The NIC raises an interrupt, unless notification is being delayed or the driver is already polling the queue. The handler acknowledges the event and arranges for packet-processing work to run.

6. The kernel consumes and replenishes

The driver examines completed descriptors, synchronizes the received data for CPU access when the platform requires it, and passes packets into the networking stack. It then supplies replacement buffers so the NIC can continue receiving traffic.

The repeating ownership cycle is:

Ownership returns to where it started. A driver that stops supplying replacement buffers breaks the loop, and the controller runs out of places to put arriving packets.

Ownership matters because the CPU and device must not modify the same descriptor or buffer at the same time without an agreed protocol.

Storage controllers use a similar pattern: the driver submits descriptors containing buffer addresses and block operations, the controller transfers data through DMA, and completion entries tell the driver which requests finished.

A process uses virtual addresses, but a device cannot safely receive an arbitrary process pointer and begin reading memory. Devices need addresses valid in their own DMA view, and the kernel must control which memory they can access.

The driver uses the operating system's DMA-mapping interface. That interface produces a device-visible address and performs any platform-specific setup.

On systems with an I/O memory management unit, or IOMMU, the device can use an I/O virtual address. The IOMMU translates that address to physical memory and checks whether the device is permitted to access the mapping:

The device issues DMA to an I/O virtual address, the IOMMU translates and checks it, and the access lands on a physical RAM page.

This provides isolation against accidental or malicious DMA outside approved buffers. It can also present scattered physical pages through a convenient device address space.

Systems without an IOMMU rely on other platform and driver constraints. In either case, the kernel—not an untrusted application—normally establishes the DMA mapping.

Scatter-gather DMA

A large logical buffer is not always one physically contiguous region. Scatter-gather DMA lets the driver describe a list of memory segments that the controller processes as one logical transfer.

The controller follows the list and transfers the segments as one logical operation. This reduces the need to copy data into one contiguous temporary buffer solely to satisfy the device.

Scatter-gather reduces extra copying, but it does not mean that all I/O becomes zero-copy. Data may still move between application memory and kernel buffers elsewhere in the complete I/O path.

DMA places another actor beside the CPU in the memory system. The device may read memory that the CPU recently modified, or the CPU may read memory that the device recently filled.

Two hazards must be prevented:

Some platforms provide hardware-coherent DMA, where cache-coherence mechanisms handle much of this coordination. Other platforms require explicit cache maintenance. Even on coherent platforms, ordering and ownership rules still matter.

Drivers use the operating system's DMA API rather than making assumptions about cache behavior. The API communicates the transfer direction and provides synchronization operations when required by the architecture.

This is also why descriptor ownership changes are carefully ordered. The driver must finish writing a descriptor before telling the controller that it owns the entry. On completion, the driver must observe the controller's status before consuming the associated data.

Interrupts are efficient when events are infrequent. The CPU does useful work or enters a low-power state, and the device interrupts only when attention is needed.

At very high event rates, one interrupt per completion becomes expensive. Each interrupt changes the CPU's execution path, runs handler code, and can disturb caches. A fast NIC might complete many packets in less time than it takes to handle separate interrupts for each one.

Systems use several techniques to balance latency and CPU cost.

Interrupt coalescing

A controller can wait for several completions, or for a short timer, before raising one interrupt. The driver handles the batch together.

Coalescing reduces interrupt rate and improves throughput, but it can add delay to the first completion in a batch.

Polling under load

A driver can temporarily inspect a completion queue repeatedly instead of accepting an interrupt for every new entry. Polling consumes CPU time, but it can process a busy queue efficiently and avoid repeated interrupt transitions.

Linux network drivers commonly use a hybrid strategy:

This approach uses interrupts when traffic is sparse and batching when traffic is heavy.

Multiple queues and CPU affinity

High-throughput devices often provide multiple command or receive queues. Separate interrupt vectors can route those queues to different CPUs.

Good placement spreads work without causing unnecessary movement of queue data between CPU caches. Poor placement can overload one core even while other cores remain mostly idle.

The correct choice is workload-dependent:

None is universally fastest.

Linux exposes enough information to connect the model to real hardware without writing a driver.

Identify a controller and its driver

List PCI devices and their bound kernel drivers:

A network controller entry may resemble:

The hardware is the controller. example_nic is the kernel driver operating it.

To inspect one device more closely, use its PCI address from the first column:

Depending on the device and permissions, the output may show memory regions used for MMIO and capabilities such as MSI or MSI-X.

Watch interrupt counters

Linux reports interrupt activity in /proc/interrupts:

A simplified multi-queue device may appear as:

The names and layout depend on the hardware, driver, kernel, and execution environment. Physical machines usually expose more useful controller detail than containers or some virtual machines.

To watch counters change:

Generate network or storage activity and look for counters associated with the relevant driver. If several queue counters rise on different CPUs, the device is distributing completion work. If one counter rises extremely quickly, the device may be busy, but interrupt count alone does not reveal how many packets or storage requests were processed because one interrupt can represent a batch.

A device driver is kernel software that translates operating-system requests into a controller's hardware protocol. The controller exposes registers or queues for command submission, status, and completion.

Programmed I/O makes the CPU transfer data through controller registers. Interrupt-driven I/O lets the CPU run unrelated work until the controller reports an event. DMA goes further by letting the controller transfer bulk data directly between the device and main memory after the driver establishes safe buffers and mappings.

DMA requires explicit buffer ownership, device-visible addresses, correct cache synchronization, and completion handling. IOMMUs can translate and restrict device memory access, while scatter-gather DMA lets one request span several physical memory regions.

Interrupts work well for sparse events; polling and coalescing reduce overhead under high load. Modern devices combine DMA, multiple queues, batched completions, and routed interrupts to move large amounts of data without making one CPU handle every byte or every completion separately.

The central mental model is:

The CPU controls I/O, the controller carries it out, DMA moves the bulk data, and interrupts report when the kernel should pay attention.