Why Thread Pools Exist and How to Size Them

Imagine a server suddenly receives 10,000 requests. One approach is to create 10,000 threads so every request can start immediately.

That sounds fast, but it removes any meaningful limit on concurrency. Each thread consumes memory and operating-system resources. Thousands of runnable threads compete for a limited number of CPU cores, while thousands of blocked threads can overwhelm databases and other downstream services.

A thread pool solves this by putting a limit on how much work can run at once:

A thread pool maintains a reusable set of worker threads and assigns tasks to them through a queue.

This separates the number of tasks waiting to be processed from the number running concurrently. You can receive 10,000 requests without creating 10,000 kernel threads.

The real challenge is choosing the right pool size and deciding what should happen when tasks arrive faster than the pool can process them.

This chapter explains why pools bound concurrency, develops sizing models for CPU-bound and blocking work, and examines queue capacity, downstream limits, overload policies, observability, tuning, and shutdown.