A custom iterator class with __iter__ and __next__ works, but the bookkeeping is painful: a state attribute, a manual StopIteration, a fresh class for every new iteration pattern. Generators replace almost all of that with one keyword. A function that contains yield automatically becomes a generator function, and calling it gives you back an iterator with the same iterator protocol, packaged in three lines of code instead of twenty.

A regular function runs from top to bottom every time you call it and produces a single return value. A generator function looks like a regular function with one twist: somewhere inside its body, it uses yield instead of (or in addition to) return. The presence of yield changes everything about how Python treats the function.

Two things make this different from a normal function. First, calling read_orders() does not run any of the body. It builds and returns a generator object instead, with the function's body parked on the very first line.

Second, when you eventually pull values out, the body runs one yield at a time. Each yield hands a value back to the caller and pauses the function with all its local variables intact. The next pull resumes from exactly that point.

That StopIteration at the end is the standard iterator signal. Python raises it automatically when the generator function returns (either by falling off the bottom or by hitting a bare return). You did not write a single if, you did not increment a counter, you did not raise an exception manually. The generator handled all of it.

To prove the function really did suspend mid-body, drop a print between yields:

Notice that the first print from inside the function only ran when we called next(gen), not when we called read_orders(). The function body is dormant until something asks for a value.

The iterator protocol contract: an object is iterable if it has __iter__, and that method returns an iterator (an object with __next__). A generator object satisfies the contract for free. It implements both methods, and __iter__ returns the generator itself.

That iter(gen) is gen line is the key tell of an iterator. Calling iter on a list returns a fresh iterator each time. Calling iter on a generator returns the same generator. There is only ever one cursor through a generator's values.

Because a generator is an iterator, you can use it everywhere an iterator works. A for loop, the list constructor, tuple unpacking, the in operator. Python just calls __next__ until StopIteration shows up.

Each of those three uses called read_orders() again, which built a fresh generator object. The body of the function ran independently each time.

What happens at each yield is the part worth slowing down on. The generator function has its own stack frame, just like any function call. When yield runs, Python freezes that frame: every local variable, the value of any open loop counter, even the exact bytecode instruction to resume at. The frozen frame stays alive inside the generator object. When next is called again, Python thaws the frame, picks up at the instruction after yield, and runs forward until the next yield, return, or end-of-function.

The same generator can have arbitrary code between yields, including loops, conditionals, and exception handling. All of it suspends and resumes correctly.

The total variable survives across yields. So does the loop's position in prices. The first call to next runs the function up to the first yield, pausing inside the for loop. The second call resumes inside that same loop, increments total, and yields the new snapshot. Nothing was re-initialized between calls.

This is what makes generators useful for streaming: you can compute a value, hand it back, and pause until the caller actually needs the next one. Memory is bounded because only one item exists at a time.

The diagram traces the back-and-forth between the caller and a generator. Each next resumes the frozen frame, runs forward until the next yield, and parks again. When the function falls off the end, Python raises StopIteration to tell the caller there is nothing left to deliver.

You never raise StopIteration by hand inside a generator. Python does it for you the moment the function returns. There are three ways a generator function can return:

In all three cases, the generator object's next __next__ call raises StopIteration.

A bare return is useful when you want to stop early, for example when a search hits the item you wanted:

The return runs after yielding "pen". The generator stops there. The lamp is never visited because the function exited.

return value Attaches to StopIteration

A regular return value inside a generator does not yield that value. Instead, Python attaches it to the StopIteration exception as StopIteration.value. You can catch the exception and read it.

This is a niche pattern in everyday code. A for loop swallows StopIteration without exposing stop.value, so most callers never see it. The feature exists mainly for coroutine-style generators and for yield from. For now, just know that return value is legal in a generator and the value rides along on the terminating exception, it does not get yielded.

A custom iterator class to walk a list of products one at a time looks roughly like this:

That class has four parts: an __init__ to store state, an index attribute, an __iter__ that returns self, and a __next__ that manages bounds and raises StopIteration. Every new iteration pattern means another class with the same scaffolding.

The generator version replaces the whole class with a function:

Three lines instead of fifteen, and the behavior is identical. No index, no bounds check, no StopIteration to raise. The for product in products loop owns the cursor, and Python turns the function into an iterator on your behalf.

The same simplification scales to harder iterators. A paginator written as a class looks something like this:

Rewritten as a generator function, the whole paginator collapses into:

Same protocol, same StopIteration at the end, a fraction of the code. Most of the time when you reach for a custom iterator class, you actually want a generator function.

Every generator object is always in one of four states. Knowing them by name helps when reading other people's code and explains the error messages you see when you misuse a generator.

The inspect module from the standard library lets you peek at the current state:

GEN_RUNNING is rare in practice because you can only observe it while the generator is mid-step, which usually means from inside the generator itself (or another thread).

The state diagram captures the full lifecycle. A generator is born in CREATED, ping-pongs between RUNNING and SUSPENDED for each yield, and eventually lands in CLOSED, either by returning normally or by being shut down. Once it's in CLOSED, there is no way back.

A generator object is single-use. Once it raises StopIteration, every subsequent next call also raises StopIteration. The function body does not restart. You have to call the generator function again to get a fresh generator.

The first list(g) drained the generator. The second one ran iter(g) (which returned the same exhausted generator) and immediately got StopIteration, so it produced an empty list. This catches people often, especially when passing a generator into two consumers that both expect to scan it.

The second loop body never ran. The fix is either to call order_stream() again or to materialize the values into a list first if you need to iterate twice:

Materializing the list loses the memory benefit, of course. If you genuinely need to walk the values twice without paying that cost, call the generator function twice.

A common e-commerce task is to walk a list of order records and produce a summary string for each one. With a generator, the loop body decides when to read the next record. Memory stays bounded even if you have millions of orders, because only the current order is held at a time.

Each summary is built only when the for loop asks for it. If you wanted just the first one, you could pull next once and stop, and the rest of the orders would never be touched.

This laziness is the second big reason generators win over building a list up front: you only pay for what you consume.

A generator can also carry useful state across yields. Here's one that yields the running total of a shopping cart as each item is added:

The local variable total survives across pauses. Writing the equivalent as an iterator class would need a self.total attribute, a self.index, and explicit bounds management. The generator hides all of that inside Python's frame machinery.

The paginator from earlier deserves a closer look because it shows how a generator can yield non-trivial values (lists of items, in this case) without the caller having to know anything about the internal index.

The generator owns the index. The caller just iterates. Compare this to a class-based version, which needs an __init__, an __iter__, a __next__, and a self.index attribute. Same behavior, four times less code.

Generators also compose well. Here's a generator that filters paginated products by stock, layered on top of paginate:

in_stock_pages is itself a generator. It pulls from paginate one page at a time and yields only the pages that have at least one in-stock product. Neither function ever holds the full catalog in memory beyond a single page.

Generators have three extra methods that come up rarely in everyday code but are worth knowing exist. They turn a generator into something closer to a coroutine: a function the caller can communicate with mid-execution. The full pattern is mostly used by frameworks (older asyncio code, contextlib decorators); the rest of this section sticks to plain yield and next.

g.send(value)

When a generator's yield is on the right side of an assignment, yield can also receive a value from the caller. The caller delivers it with g.send(value), and the generator picks it up as the result of that yield expression.

The first call has to be next(g) (or equivalently g.send(None)) to run the function up to the first yield. After that, each send resumes the generator with the sent value as the result of yield cart. This bidirectional behavior is the seed of how async/await was originally built on top of generators, but it's overkill for most data-streaming work.

g.throw(exc)

g.throw(SomeError) resumes the generator by raising SomeError at the current yield point. The generator can catch it with a normal try/except inside its body. This is mainly useful for tearing down resource-holding generators.

g.close()

g.close() is the polite shutdown. It raises GeneratorExit inside the generator at the current yield. If the generator catches it (in a try/finally for cleanup, for example), it can release resources before returning. After close, the generator is in GEN_CLOSED.

close is what with blocks effectively do to generators they own through contextlib.contextmanager. Treat these three methods as advanced tools; the rest of this section assumes plain next-driven generators.

A for loop over a generator is the same protocol you'd run by hand. It calls iter on the object (which returns the generator itself), then repeatedly calls next until StopIteration shows up. The loop swallows the exception and exits.

A generator is just a particularly easy way to satisfy the iterator protocol. Everything that accepts an iterable accepts a generator: for, list, tuple, set, sum, max, sorted, any, all, in, and tuple unpacking.

Each call rebuilds the generator from scratch, which is why calling stock_counts() four times all give correct results. If we'd assigned the generator once and reused it, only the first call would have seen the values.

The single biggest source of confusion with generators is forgetting they're one-shot. A function that takes a generator and walks it twice will see the values once and emptiness the second time. Here's the bug in a paginator helper:

What's wrong with this code?

The first line of report walks the entire generator to count. The second line tries to read the first value, but the generator is already exhausted. next(iter(...)) raises StopIteration, which then bubbles out of report.

Fix:

The fix is to materialize the generator into a list at the top of the function. You lose the streaming memory benefit, but you can now scan the data twice. The alternative is to redesign report to make a single pass: keep a running count and remember the first value as it goes by.

It helps to see the same logic in three forms: a list built up front, a custom iterator class, and a generator function. All three produce the same values, but with very different cost profiles.

The generator gives you the laziness of an iterator class with the brevity of a list comprehension. The cost is the one-shot constraint. For most stream-processing work, that's an excellent trade.

The list version actually does five separate sum calls, each one scanning a slice of prices. The generator threads the running total through and visits each price exactly once. Same output, much better complexity if the input is large.

To put the chapter's main point in one place, here is the same paginator written twice. The class form and the generator form, side by side.

Custom iterator class:

Generator function:

Both print the same output:

The class has four explicit pieces of state machinery. The generator has none of them. As you read more Python code in the wild, you'll see this pattern repeat: a generator function in place of what would otherwise be a small iterator class. The class form is still useful when the iterator needs richer behavior (multiple iteration entry points, attached methods, subclassing), but for "produce this stream of values," the generator wins.