Spawning a Process per task gets ugly fast. If you have 10,000 product images to resize, you don't want 10,000 processes, you want a small crew of workers that pick up one image after another until the queue is empty. multiprocessing.Pool is that crew. It manages a fixed group of worker processes, distributes tasks across them, collects results, and shuts everything down cleanly. This lesson covers the API, the dispatch patterns (map, imap, apply_async), how chunksize affects throughput, error propagation, and the lifecycle methods you need to clean up correctly.

When you create a Process per task and call start() and join() on each one, That works for a handful of tasks. Once you have hundreds or thousands, three problems show up:

Pool solves all three. You create it once with N workers (where N is usually os.cpu_count()), hand it an iterable of inputs, and it dispatches work across the existing workers. The same N processes handle everything. Worker reuse means you pay the startup cost once.

The dispatcher hands tasks to whichever worker is free. Workers run independently in their own processes, so they bypass the GIL entirely. When all tasks are done, results come back to the parent in the order you asked for (or in completion order, depending on the method you call).

The constructor is straightforward:

The processes argument sets how many worker processes the pool will keep running. If you leave it out, Pool defaults to os.cpu_count(), which is the number of logical cores on your machine.

For CPU-bound work like image processing or hashing, os.cpu_count() is usually the right answer. More workers than cores doesn't add speed, just overhead. For I/O-bound work, you might use a higher number, but for I/O Pool isn't the right tool anyway. asyncio or threads work better there.

The pool starts its workers immediately when you create it. Each worker is a separate Python process that sits idle, waiting for work to arrive. You don't need to start them yourself.

The most common method is pool.map(func, iterable). It looks like the built-in map, but it runs the function in parallel across the pool's workers and returns a list of results in the same order as the input.

A few things to notice:

pool.map blocks until all results are ready. The parent process sits there waiting. For most batch jobs, this is what you want, you just need the answers and don't care about progress reporting.

pool.imap(func, iterable) does the same work as map, but returns an iterator instead of a list. Results come back one at a time, in input order, as workers finish them.

The output order matches the input order. The difference from map is that you can start processing each result as it arrives, without waiting for the whole batch. That matters when:

pool.imap_unordered(func, iterable) is the same idea but gives up the ordering guarantee. Results come back in whatever order workers finish them. This is faster because a worker doesn't have to wait for an earlier task to finish before its result can be yielded.

Use imap_unordered when you don't care which input produced which result, only that they all get processed. A common pattern is when each result is self-contained, for example, "resize this image and save it with a known filename." The output filename is derived from the input, so you don't need to know which task finished first.

The three methods cover most batch dispatch needs. Pick map when you want a list back and don't need progress. Pick imap when you want a progress indicator or lazy input. Pick imap_unordered when ordering doesn't matter and you want the best throughput.

map and imap are good for "run this function over a batch." apply_async is the lower-level primitive: it schedules a single function call and returns an AsyncResult object that you can query later.

apply_async(func, args, kwds) queues one call. It returns immediately with an AsyncResult. The actual call happens in a worker. To retrieve the answer:

map_async(func, iterable) is the async version of map. It schedules the whole batch and returns one AsyncResult that yields the list of results when ready.

The _async variants let the parent process do other things while the pool works. A common pattern is to launch several map_async calls against different datasets and then collect their results in any order.

All three calls run in parallel. The get() calls block in order, but the work was already happening concurrently in the background. This pattern is useful when you have a fixed set of independent jobs and want fine-grained control over when to wait for each one.

A callback parameter lets you register a function to run when a result is ready, without polling:

The callback fires in the parent process, in the thread that handles result delivery. Keep callbacks short and non-blocking.

The map, imap, and imap_unordered methods all accept an optional chunksize argument:

chunksize controls how many items the pool sends to a worker in one batch. Default is 1 for imap and imap_unordered. For map, the pool picks a default based on the input length, roughly len(iterable) / (processes * 4).

Why does this matter? Every item you dispatch to a worker has a per-item cost: pickling the input, sending it through a pipe to the worker, unpickling it on the other side, pickling the result, sending it back, unpickling. If each individual task is tiny (microseconds of CPU work), this dispatch overhead can dominate.

With chunksize=1, the pool dispatches each integer one at a time. The actual work, adding one to a number, takes nanoseconds. The dispatch overhead is orders of magnitude larger. With chunksize=10000, each worker pulls 10,000 items at once, processes them in a tight loop locally, and sends back the batch. The per-item cost amortizes.

The trade-off works in both directions:

The other consideration is load balancing. With a very large chunksize, if one worker gets a chunk with all the slow tasks and another gets a chunk with all the fast ones, the slow worker holds up the whole pool. Smaller chunks let work redistribute as workers finish.

For our image-resize example coming up, each task takes tens of milliseconds, so a chunksize of 1 or a small number is fine. For trivial per-element transforms over millions of items, crank it up.

A pool has three lifecycle methods you'll see in real code:

The typical pattern without a context manager looks like this:

Forget either close() or join(), and you'll leak worker processes. They'll keep running after your program exits, holding file descriptors and memory.

The clean way is to use Pool as a context manager, which is what we've been doing in most examples:

The with block automatically calls terminate() when it exits (whether normally or because of an exception). This is convenient for scripts, but be aware: terminate() is the abrupt variant. If you want workers to finish in-flight tasks before shutdown, do it explicitly:

For most batch jobs that finish all their work via map before the with block ends, this distinction doesn't matter, all the work is already done by the time terminate() runs.

The three shutdown paths cover the cases you'll encounter. Graceful drain when you want to let work complete, hard stop when you don't, and the context manager default for short-lived scripts.

A worker that raises an exception doesn't crash the whole pool. Instead, the exception is captured, pickled, sent back to the parent, and re-raised when the parent tries to retrieve the result.

The parent sees the same exception type and message that the worker raised. The traceback even shows where in the worker function it happened. This is convenient because you don't need a special "worker exception" type, normal try/except works.

The catch is that map stops yielding results once any task raises. If three out of four tasks succeed and one fails, you lose the three successful results. If you want to collect successes and failures separately, use apply_async per task and check successful():

Now each task's outcome is tracked independently. Successes give results, failures give exception details, and the pool keeps running.

One more thing: the exception must be picklable. Most built-in exceptions are, but custom exceptions with unpicklable arguments (open file handles, lambdas, etc.) can't be sent back. The worker will then raise a different error during the pickling step, usually a confusing one. Keep your exception types simple.

Time to put it all together. An e-commerce site uploads new product images. Each image needs to be resized into three sizes (thumbnail, medium, large) and saved to disk. The total job: 200 images, 3 sizes each, on an 8-core machine.

We'll simulate the resize work with a CPU-bound function so we don't depend on an image library. The pattern is identical with a real library like Pillow.

The speedup isn't a perfect 8x because some tasks (the large 1200-pixel ones) take longer than others, and the OS scheduler doesn't put every worker on its own core all the time. Real-world parallel speedups on CPU-bound work tend to land between 60% and 90% of theoretical, which is what we see here.

A few things worth pointing out in this code:

For real image processing with Pillow, the worker function would look like:

The pool dispatch code stays exactly the same. The pool doesn't care what your worker function does; it just hands it inputs and collects outputs.

Now let's add progress tracking with imap_unordered. For a long-running batch, you want to know it's making progress.

The imap_unordered loop gives us per-result granularity. The order isn't guaranteed, but each filename ends up with its own output, so we don't care which workers finished first.

A second concrete case: nightly export of order data to a CSV file. Each order has line items, and computing a per-order summary (totals, tax, etc.) is CPU work. We have tens of thousands of orders.

The speedup is lower than the image example because each order is light work; dispatch overhead eats more of the gain. The chunksize=200 mitigates this. If we used the default chunksize=1, the timing would actually be worse than serial.

Try it yourself: drop chunksize=200 from the map call and rerun. On most machines, the parallel version becomes slower than the serial one. Then bump it to 1000 or 2000. You'll see the parallel version pull ahead.

This is the practical rule for tuning: if a single task is light, raise chunksize until throughput stops improving. The crossover point varies by task and machine, so measure rather than guess.

A common operation in storage systems is verifying file integrity by computing checksums. For an e-commerce site with thousands of product images stored on disk, you might run a nightly verification job that recomputes hashes and compares them against a database.

Hashing is purely CPU-bound (assuming the file fits in OS cache), which makes it a near-ideal pool workload. Each task is heavy enough that dispatch overhead doesn't matter, and there's no shared state between tasks.

The multiprocessing.Pool API is the original one, and it works. There's also concurrent.futures.ProcessPoolExecutor, which is a newer wrapper that many people prefer because it uses the same submit / map / Future API as the thread pool variant. The underlying machinery is the same; the API is just nicer in places.

For now, knowing multiprocessing.Pool directly is valuable because:

If you're starting a new project from scratch, ProcessPoolExecutor is often the better choice. For everything in this lesson, you can apply the same concepts there.