A pipeline is a chain of stages where each stage is a goroutine that reads from an input channel, does some work, and writes to an output channel. The shape lets you compose concurrent transformations the same way you'd chain function calls, but with each stage running on its own goroutine so they all make progress at the same time. This lesson covers the canonical pipeline shape, how stages close their outputs, how cancellation works through a shared done channel, and how to add fan-out inside a stage when one step is the bottleneck.

A pipeline is a directed chain. Each stage owns one input channel and one output channel. It reads values from its input, transforms them, and writes results to its output. The next stage reads what this stage produced, and so on, until a sink at the end does something terminal (prints, writes to a database, returns a count).

Go pipelines follow a small convention: a stage is a function that takes its input channel and returns its output channel. Every stage has the same shape, give or take element types.

The input is <-chan T (receive-only, upstream element type). The done channel is <-chan struct{}, a shared cancellation signal every stage watches. The return type is <-chan U, so the next stage can read from it but can't accidentally send into it.

Because each stage returns its output channel, you compose stages by passing one through the next:

Three function calls, three goroutines running concurrently, one consumer loop.

The solid arrows show data flow. The dashed arrows show done reaching every stage as a shutdown broadcast. Each stage closes its own output when it's done, which signals the next stage to wrap up.

This is distinct from fan-in, which is a single merge step combining several channels into one. Pipelines focus on staged transformation, where each stage turns one stream into another. You can use fan-in inside a pipeline (when one stage runs N workers, fan-in merges their outputs back), but the pipeline itself is the chain, not the merge.

Pipelines come into focus when you see one running. Here's a complete e-commerce example: stage 1 generates raw orders, stage 2 enriches each order with product details, stage 3 filters out totals below a threshold. The main goroutine is the sink.

Each stage takes the previous stage's output as input and returns its own output. The composition in main reads top to bottom: orders flow from generate into enrich into filter and out. Nothing in the function bodies knows where it sits in the chain; you could slot in a new stage without touching either neighbour.

Each stage uses defer close(out) and a for-range over its input. The range exits when the input channel closes, which signals that the previous stage is done. The deferred close then fires, signalling the next stage. That cascade is the drain-on-empty behaviour.

Each out <- v is wrapped in a select with <-done. If the consumer stops reading and the program calls close(done), every stage's select picks the <-done case and returns.

The closing rule for pipelines is one sentence: each stage closes its own output channel when its input drains.

That single rule keeps the pipeline safe. Every stage closes the channel it returns. No stage closes another stage's channel. No stage closes its own input.

Why this rule? Two reasons. First, in Go a channel must be closed by the sender, never the receiver. If a receiver closed it, the sender would panic on its next send. So the closing duty has to live with whatever goroutine is sending. Second, closing must happen exactly once; assigning each channel to exactly one owner gives that for free.

Each stage owns the channel it returns. "Owns" means: created it, sends on it, and closes it when done. The downstream stage only reads. The main goroutine never closes anything except the done channel.

Here's a quick table that captures the responsibilities. The pattern repeats for every stage in any pipeline.

The sink doesn't close anything because it doesn't send. It just reads until its input channel closes, then exits the for-range. That's how the pipeline ends cleanly.

Pipelines get backpressure for free. The channels between stages are the backpressure mechanism.

If a downstream stage runs slowly, the upstream stage's out <- v blocks until the downstream reads. That block propagates: the upstream's for-range over its own input doesn't advance, so its upstream stays blocked too. Eventually the generator stops emitting because every send is blocked. The whole pipeline runs at the speed of the slowest stage. No queues grow unbounded.

This is usually what you want. If you have a fast generator and a slow database writer at the end, the database writer's pace controls the whole chain.

Unbuffered channels give the tightest backpressure: every send waits for a paired receive. Small buffers (1 to a few dozen) smooth out short bursts where one stage is briefly faster than the next. Large buffers defeat the purpose; they let pressure build up in memory, which is the thing backpressure exists to prevent.

Start with unbuffered. Add buffering only when a brief burst at one stage would otherwise pin the upstream stage for no good reason. If the pipeline feels too slow, buffering won't help; either make the slow stage faster or run several copies of it in parallel.

A pipeline must shut down cleanly when the consumer stops caring. If the sink exits early (the user hit Ctrl-C, an error fired, a match was found), upstream stages need to know. Otherwise they park forever trying to send into channels nobody reads.

The standard fix is a shared done channel. Every stage takes done <-chan struct{} and watches it in any select where it might otherwise block:

If the consumer closes done, the <-done case in every stage's select fires immediately. Each stage returns, triggers its deferred close(out), and the next stage's for-range exits when it sees the close. The cascade runs from the back of the pipeline forward.

In the running example, main calls defer close(done) right after creating the channel. That close runs when main returns, no matter how. Normal completion and early exit (panic, break, error) both run it. Every stage sees it and shuts down.

A single close(done) reaches every stage at once. Each stage returns, runs its deferred close(out), and the pipeline tears down.

The read side is usually handled by the cascade itself: for o := range in exits when the input closes, and the input closes because the upstream stage also watches done. If a stage blocks on something other than its input, add an explicit <-done case there too.

The context.Context API formalizes this pattern with deadlines, values, and call-tree propagation. For pipelines that just need "stop now" with no value passing, a plain done chan struct{} is enough.

Cancellation looks right on paper and goes wrong in practice. Here's a pipeline that runs cleanly when the consumer drains everything, but leaks goroutines if the consumer quits early.

What's wrong with this code?

The output looks fine, but two goroutines are now leaked.

After the consumer reads two values and exits, the double goroutine sends one more value onto out, then tries to send the next. That send blocks forever because nobody is reading anymore. The generate goroutine is parked on the channel between generate and double, because double is no longer reading. Both goroutines stay alive for the rest of the program. In a long-running service that builds a fresh pipeline per request, this leaks two goroutines per request until the process runs out of memory.

Fix: Thread a done channel through both stages, wrap each send in a select, and defer close(done) in main.

Same observable output. The difference is that when main returns, close(done) fires. Every stage's select notices and returns. No leaked goroutines.

The line defer close(done) is small but load-bearing. Every pipeline that lets the consumer exit early needs a guaranteed way to broadcast cancellation. A deferred close in the goroutine that owns done (almost always main or the request handler) is the simplest way to get that guarantee.

Most stages run at the same speed as their neighbours. But sometimes one stage is the bottleneck, like a call to an external API or a CPU-bound computation. The fix is to parallelize that stage: run N worker goroutines that all read from the same input channel. Each worker processes one item at a time, and together they process N items in parallel. The workers' outputs feed a fan-in step that merges everything back into a single channel.

Here's enrich from the earlier example, parallelized with three workers. Suppose priceOrder now takes 20ms (simulated with time.Sleep). One worker would handle six orders in 120ms; three workers do it in roughly 40ms.

The rest of the program (types, catalog, generate, filter, main) stays the same. In main you swap enrich(done, raw) for enrichParallel(done, raw, 3). The output looks similar to before, but order numbers no longer appear in source order because the three workers race to process them.

The shared input channel acts as a load balancer: when a value lands on in, the runtime hands it to whichever worker is ready. The signature of enrichParallel is identical to a single-worker stage, so the rest of the pipeline doesn't know parallelism is happening inside. The done channel reaches both workers and the fan-in goroutines, so cancellation still cascades correctly.

Ordering inside this stage is gone. If a downstream stage needs ordered output, keep this stage single-threaded.

Errors in a pipeline don't fit the same shape as values, but they have to go somewhere. The cleanest approach for most pipelines is to piggyback errors on the value type. Each pipeline value carries a successful payload or an error, so a single channel handles both:

The enrich stage emits an enrichedOrder for every input. If processing succeeds, err is nil. If it fails, err is set and order is the zero value. Downstream stages and the consumer check err before using order.

Two other options come up. Log-and-drop is the simplest: skip the item and write the error to a log. The earlier enrich example used this. It's fine when nobody downstream needs to know which items failed. A separate error channel works too: stages send successes on the main output and failures on a dedicated errs chan<- error. That keeps the success channel clean of failures, at the cost of plumbing a second channel through every stage.

For most pipelines, piggyback is the right default. It keeps each value paired with its outcome and doesn't add extra channels.

A handful of mistakes show up repeatedly in pipeline code. None are caught by the compiler.

Forgetting to close the output. A stage that never closes its output channel hangs every downstream stage forever. Always pair the owning goroutine with defer close(out).

Double-closing a channel. A stage that tries to close its input panics with close of closed channel. One closer per channel, and the closer is always the sender.

Sending without watching done. A naked out <- v leaks the goroutine if the consumer goes away. Wrap every send in a long-lived stage in a select with <-done.

Missing done broadcast. If main returns without closing done, every stage parked on <-done stays parked. defer close(done) right after creating the channel guarantees the broadcast.

Cycles. A pipeline must be a directed acyclic graph. If stage B's output feeds back into stage A's input, backpressure can deadlock the whole loop. Cycles need a different pattern (event loops, work queues with explicit deduplication).