A thread pool is a fixed (or bounded) set of worker threads that pull tasks from a shared queue. Instead of spawning a brand-new thread every time you've work to do, you submit the work as a task and let one of the already-running workers pick it up. This lesson covers why thread creation hurts when you do too much of it, how a pool fixes that, the four pool flavors that Executors hands you out of the box, how to size a pool for CPU-bound vs IO-bound work, and the trade-offs you sign up for when you drop down to ThreadPoolExecutor. The Executor API itself, Callable and Future, CompletableFuture, and the virtual-thread executor each get their own chapter; here we focus on the pool concept.

A Thread looks cheap in your code. You write new Thread(task).start() and you're done. A lot is happening behind that one line, and most of it costs real money.

Each Java thread is backed by an OS-level thread. Creating one means asking the operating system to allocate a stack (Java's default is around 512 KB to 1 MB per thread on most platforms), register the thread with the kernel scheduler, and wire up some bookkeeping inside the JVM. On modern hardware, that ballpark is a few tens of microseconds and a chunk of memory you don't get back until the thread dies. A few threads, fine. A few thousand threads, every one of them eating a megabyte of stack, and you've burned a gigabyte of RAM before you've done any actual work.

The kernel scheduler is the other half of the bill. When you've eight CPU cores and you've spawned eight hundred threads, the scheduler still has to give each thread its turn. It does that by context switching: saving one thread's CPU state to memory, loading another thread's state, and resuming execution. Each switch is on the order of a few microseconds. Multiply by hundreds of thousands of switches per second and the CPU spends more time juggling threads than running your code.

The diagram shows where the costs come from. Your new Thread().start() triggers a kernel-level allocation, the stack memory sits around until the thread exits, and once you've more threads than CPU cores the scheduler is burning cycles to rotate them. None of this is visible from your code, which is the trap.

The third cost is a hard cap. Operating systems put a limit on how many threads a single process can hold. On Linux that's usually a few thousand by default, governed by /proc/sys/kernel/threads-max and the process's ulimit. The JVM doesn't get its own bigger pool. Cross the limit and Thread.start() throws OutOfMemoryError: unable to create new native thread, which is a misleading message because the actual problem is "the OS said no". Programs that spawn an unbounded number of threads tend to crash here, often in production, on a busy day.

A single thread takes around 1 MB of stack plus a few tens of microseconds to start. Creating one per request in a service that handles thousands of requests per second is a fast path to running out of memory or hitting the OS thread limit.

The exact number depends on the OS, the JVM flags, and the available memory, but the point holds: a tight loop creating threads doesn't get to keep running. Somewhere between a few thousand and a few tens of thousands, the JVM refuses to make more. Pool-based code, doing the same workload, would stay flat at whatever pool size you picked.

A thread pool fixes the cost problem by reusing threads. You decide up front how many workers you want (or set a cap), the pool creates that many threads once, and from then on you hand it tasks instead of threads. Each worker sits in a loop: pull the next task off a shared queue, run it, repeat.

The two moving parts are the workers (a small set of long-lived threads) and the task queue (a thread-safe queue that holds work waiting to be picked up). When you call submit(task) or execute(task), the pool puts your task on the queue. The first idle worker pulls it off and runs it. When it finishes, the worker doesn't die; it goes back for the next task.

The diagram is the entire model. Code on the left hands tasks to the queue. Workers on the right pull from the queue. The queue is the buffer that decouples "how fast tasks arrive" from "how fast workers can finish them". If tasks arrive faster than workers can drain them, the queue grows. If workers run out of tasks, they block on the queue and wait. The shape doesn't change.

This setup wins on three fronts. First, thread creation happens once, at pool startup, so the per-task cost drops to "put something on a queue", which is fast. Second, the number of OS threads stays bounded by the pool size, so you can't accidentally spawn ten thousand of them. Third, the queue gives you a place to absorb bursts: if a hundred requests arrive in the same second and you've eight workers, ninety-two requests wait in the queue instead of spawning ninety-two new threads.

The trade-offs follow from the same shape. If the queue grows without bound and tasks arrive faster than they can be processed, you'll exhaust memory holding tasks. If the pool is too small for your workload, requests pile up and latency climbs. If the pool is too big, you're back in the "too many threads thrashing" world. Pool sizing is the lever you tune to balance these.

The Java API for this is the ExecutorService interface, and the easiest way to get one is the factory class java.util.concurrent.Executors, which produces four flavors of pool.

Executors.newFixedThreadPool(n) gives you a pool with exactly n workers and an unbounded task queue. The workers never get added and never get retired; once the pool is built, you've the same n threads until you shut it down. New tasks always go on the queue, which grows as long as memory holds out.

This is the default pool. It's predictable, easy to reason about, and matches the typical "I have a known number of cores and a steady stream of work" pattern. The cap on threads means you won't spawn more than n, so you can't accidentally drown the CPU.

Consider an order-processing service. Each order placement triggers a small batch of work: validate the items, compute the total, write the order to the database, and email a confirmation. You don't want to spin up a thread per order during a busy hour; you want a handful of workers that grind through orders as they come in.

Four workers, six orders. The first four orders go straight to the four workers and start running in parallel. Orders five and six wait in the queue. As soon as pool-1-thread-1 finishes order 1001, it picks order 1005 off the queue. Same with pool-1-thread-2 and order 1006. No new threads were created. The same four workers handled all six tasks. The exact interleaving will vary across runs because the order in which the scheduler runs the workers isn't fixed, but the cap of four concurrent tasks always holds.

The risk of a fixed pool is the unbounded queue. If your producer (the code calling submit) is faster than the workers and you never catch up, the queue keeps growing. Each pending task is an object in heap memory, and a queue of a few million pending tasks will eventually trigger OutOfMemoryError. For most steady-state workloads that's not a real risk, but for back-pressure-sensitive systems (a busy web server hit by a traffic spike) you might want a bounded queue instead, which means dropping down to ThreadPoolExecutor directly.

newFixedThreadPool is the right starting point when you know roughly how much parallelism you want and the workload doesn't have wildly variable arrival rates. It's the safest of the four pre-built pools.

Executors.newCachedThreadPool() works differently. There's no fixed worker count. Instead, the pool creates threads on demand: if a task arrives and no worker is idle, a new thread is spawned to handle it. If a worker sits idle for 60 seconds, it's retired. The queue used internally is a SynchronousQueue, which is a queue with no capacity at all; every task has to be handed directly to a worker.

The upside is that a cached pool adapts to whatever load you throw at it. A short burst of work spawns a burst of workers, then the workers die off when the burst ends. For occasional, short-lived tasks that come in spurts, this can feel ideal.

The downside is what we spent the first section explaining: unbounded thread creation. If tasks arrive faster than they can be processed, the pool keeps spawning new threads, with no cap. A flood of slow tasks means a flood of threads, and you're back to the OS thread limit and stack-memory exhaustion.

newCachedThreadPool has no upper bound on threads. If task arrival outpaces task completion, the pool spawns workers without limit and your service follows the same crash path as raw new Thread().start() in a loop. Use it only for short-lived, low-latency tasks where you're confident the rate is bounded.

Five tasks, five different threads. Each task is short, so by the time task 2 arrives, the pool can't be sure thread 1 will be free in time, and the SynchronousQueue hand-off mechanism causes a new worker to spin up. If you wait long enough between batches, idle threads time out at 60 seconds and the next batch starts fresh.

Compare that with the fixed pool, which served the same kind of work with four threads no matter how many tasks arrived. The cached pool keeps spawning. That's fine when the burst ends. It's a disaster when the burst doesn't.

Real-world advice: avoid newCachedThreadPool for any production service that processes external input. Use it only for short scripts, internal tools, or test setups where you know what's running. For anything that touches the network or a queue you don't control, use a fixed pool (or a bounded ThreadPoolExecutor) so a misbehaving caller can't take down the JVM.

Executors.newSingleThreadExecutor() is a pool of size one. One worker, an unbounded queue, and a guarantee that tasks run sequentially, in the order you submitted them. If task 1 takes ten seconds, task 2 starts at the eleven-second mark. There's no parallelism inside the pool.

That sounds like a step back, and for raw throughput it's. The reason single-thread executors exist is serialization of access: when you've shared state and you want a series of updates applied in order without explicit locks, a single-thread executor gives you that automatically. Submit each update as a task, and they queue up and execute one after another. The shared state is touched only by the one worker thread, so synchronization simplifies to "make sure nobody else touches it".

A common use case is an event logger that writes to a single file. You don't want two threads writing to the file at the same time and producing interleaved garbage. A single-thread executor accepts events from any caller and writes them in order.

The four log lines come out in exactly the order they were submitted, even though four different callers (in a real program) could be submitting them concurrently. The single worker provides the ordering guarantee automatically. Replace it with a two-thread pool and lines would interleave unpredictably, because two workers could pick up two tasks at the same time and reach the System.out.println call in either order.

Single-thread executors are also handy when a library you're integrating with isn't thread-safe and the cheapest way to keep it safe is to confine all access to one thread. Submit every call as a task on the executor, and the library only ever sees one caller at a time.

The same caveat about unbounded queues applies: if submissions outpace the single worker forever, the queue grows without bound. For most logger-style use cases the worker keeps up fine, but it's a thing to know.

The first three pools run tasks as soon as a worker is free. A scheduled pool runs tasks at a specific time, or on a recurring schedule. Executors.newScheduledThreadPool(n) gives you n workers backed by a delay-aware queue: tasks sit in the queue with a target time, and workers only pick them up once the time has arrived.

The interface is ScheduledExecutorService, which adds three methods on top of the regular ExecutorService:

The difference between scheduleAtFixedRate and scheduleWithFixedDelay is subtle and worth understanding. Fixed rate means "run every N seconds, on the clock", regardless of how long each run takes. If a run takes longer than the period, the next run starts immediately (or back-to-back runs may overlap if n > 1). Fixed delay measures the gap from the end of one run to the start of the next, so a slow task pushes the next run later. Use fixed rate when you want predictable wall-clock timing; use fixed delay when you want a guaranteed cool-down between runs.

A classic e-commerce job is the nightly inventory report. Every night at the same time, the system snapshots the catalog, computes stock levels, and emails a report to the operations team. A scheduled pool firing once a day fits this shape well.

Three runs, one second apart, on the clock. The pool has two workers, so a longer-running task could in principle overlap, but in this example each run is instantaneous and the same worker handles successive runs (the scheduler is free to assign whichever worker is idle). The 500 ms initial delay is honored before the first run.

A second pattern is the one-shot delayed task. "Send a reminder email if the cart has been idle for an hour." You schedule the email with a 1-hour delay; if the customer checks out before then, you cancel the scheduled task.

The reminder was scheduled to fire 500 ms in the future, but we cancelled it after 200 ms. The output is the cancellation confirmation; the reminder itself never runs. ScheduledFuture is what schedule returns, and cancel is the lever for "never mind, don't run that".

newScheduledThreadPool is the go-to for periodic background jobs: cache eviction, metrics flushing, nightly reports, retry timers, anything that says "every N seconds" or "in N minutes". For one-off scheduling at a specific wall-clock time, schedule(..., delay, unit) with a computed delay (target minus now) is the standard approach.

The four pre-built pools cover most day-to-day cases. They differ in three things: how many workers they hold, what kind of queue they use, and how they react when load changes.

The decision tree is straightforward:

Many production systems avoid the factory methods entirely and construct ThreadPoolExecutor directly, because they want a bounded queue and a custom rejection policy.

All four factory methods return instances of ThreadPoolExecutor (or its scheduled subclass, ScheduledThreadPoolExecutor) with different constructor arguments. If you want full control, you can construct one yourself. The constructor takes seven arguments, and each one is a knob worth understanding.

The seven parameters, in plain language:

The flow the executor follows when you submit a task:

The decision flow runs top to bottom on every submission. Below the core size, a new worker is created right away. Once you're at the core size, new tasks go to the queue. Only when the queue is full does the pool create more workers (up to the maximum). When the queue is full and the pool is at max, the rejection handler runs. Four RejectedExecutionHandler implementations ship in the JDK:

A small bounded pool constructed by hand, with a CallerRunsPolicy for back-pressure:

The exact ordering varies, but the pattern is consistent with the flow above. The first two tasks start core workers. Tasks 3 and 4 land in the bounded queue. Tasks 5 and 6 push the pool above core, up to the maximum of 4 workers. Tasks 7 and 8 arrive while the queue is full and the pool is maxed, so the rejection handler kicks in. With CallerRunsPolicy, "main" runs the rejected tasks itself, slowing down the submitter and giving workers time to drain the queue. The end result is bounded memory use, bounded thread count, and natural back-pressure.

The factory pools are specific configurations of this constructor. newFixedThreadPool(n) is corePoolSize = maximumPoolSize = n, infinite keepAlive, unbounded LinkedBlockingQueue, default AbortPolicy. newCachedThreadPool() is core = 0, max = Integer.MAX_VALUE, 60s keepAlive, SynchronousQueue. Knowing the constructor demystifies the factories.

A LinkedBlockingQueue with no capacity argument is effectively unbounded (Integer.MAX_VALUE). If you build a ThreadPoolExecutor with that queue and don't watch your submission rate, the queue will grow until you run out of heap. Use ArrayBlockingQueue(n) or LinkedBlockingQueue(n) for a real cap.

The size of a pool isn't a guess. The right number depends on what the tasks actually do, and the answer is different for CPU-bound work (calculation, parsing, encoding) than for IO-bound work (database queries, HTTP calls, file reads). Getting it wrong by a factor of 10 is easy and common.

The starting point is the number of CPU cores you've. Java exposes this via Runtime.getRuntime().availableProcessors():

This is the number of CPU cores the JVM thinks it has. On a container with CPU limits, this often returns the CPU quota, not the physical core count. The number changes across machines, so don't hardcode it; read it at startup.

CPU-bound tasks spend their time burning CPU cycles: compressing a file, computing a checksum, running an in-memory calculation. Adding more workers than you've cores doesn't help, because the cores are already busy. Extra threads sit idle waiting for a core, and the context-switching overhead actively hurts throughput. The rule of thumb is one worker per core, or one more than the core count for a little buffer for occasional non-CPU work mixed in.

For a CPU-heavy report generator on an 8-core machine, that's 8 workers. If you set it to 32, the JVM creates 32 threads, the OS scheduler rotates them through 8 cores, and the same total CPU work takes longer than it would with 8.

IO-bound tasks spend most of their time waiting: blocked on a database response, an HTTP call, a file read. The thread is parked, not using a core, sitting on a socket. While one worker is parked, another worker could use the core to make progress on its own task. For IO-bound work, the right pool size is bigger than the core count, sometimes much bigger.

The formula people quote is Brian Goetz's:

If your tasks spend 90% of their time waiting on IO and 10% on CPU work, the ratio is 9:1, and the formula gives you cores * 10. On 8 cores, that's 80 workers. If they spend 99% on IO and 1% on CPU, it's cores * 100, or 800 workers on the same machine.

The exact number is hard to pin down without measurement. The formula gives you a starting point; production tuning is "run a load test, look at CPU usage, adjust". As a practical default for "this is a service that calls another service over HTTP and the network is the bottleneck", something like 2 * cores to 4 * cores is a common starting band.

The "pure waiting" row is also where virtual threads change the calculus. A virtual-thread executor lets you've effectively as many concurrent IO operations as you want without the OS-thread cost, so the sizing question evaporates for that style of code. For platform-thread pools (the kind this lesson covers), the formula above is what you start with.

Threads beyond the natural pool size for your workload don't speed anything up. CPU-bound work caps at the core count. IO-bound work caps at the rate the downstream resource (database, network, disk) can respond. Sizing past those limits spends memory.

This is a starting-point calculator, not the final answer. Real tuning is iterative: pick a size, run the workload under representative load, watch CPU usage, queue depth, and latency, and adjust. If CPU is pegged and latency is high, the pool's too big (or the work really is CPU-bound). If CPU is half-idle and latency is high, the pool's too small.

Pool sizing isn't a single knob with one right answer. You're balancing four things, and pushing on one usually moves another.

Throughput is how many tasks you finish per second. Bigger pools (up to a point) help throughput when work is parallel and a resource isn't saturated. Past the saturation point, more threads add overhead without finishing more work.

Latency is how long any single task takes from submit to done. Latency depends on both worker availability (no queueing when you call submit) and worker speed (not too many threads fighting for a core). Bigger pools reduce queue time but, past saturation, increase compute time per task due to context-switching.

Memory is what you pay for each worker (around 1 MB of stack) plus each queued task (the size of the Runnable object plus its captured state). A pool with 1000 workers and 100,000 queued tasks costs serious heap and stack memory, even when most workers are idle.

Back-pressure is the system's response to overload. A bounded queue plus a sensible rejection policy is what stops a busy service from accumulating millions of pending tasks. An unbounded queue defers the failure (memory exhaustion) instead of preventing it.

The interactions:

A useful framing: pick the failure mode you can tolerate. If your service can drop or reject some work during overload, use a bounded queue and a rejection policy that suits your back-pressure model. If your work is precious and can't be dropped, use an unbounded queue and budget memory for the worst case, plus monitoring that screams when the queue gets deeper than expected.

Two practical patterns:

Bulkhead. Use a separate pool for each downstream dependency. The pool that calls the payments service is sized for payments. The pool that calls the email service is sized for email. If the email service goes down and its pool fills up, the payments pool keeps running. Sharing one big pool across all dependencies means a slow dependency can starve everyone.

Bounded with caller-runs. A pool with a small core, a bounded queue, and CallerRunsPolicy gives you a graceful degradation curve. Under normal load, tasks run on workers. Under brief bursts, tasks queue. Under sustained overload, the queue fills and rejections push work back to the caller, which slows down the submission rate. The system doesn't fall over; it gets slower.

The opposite anti-pattern is what newCachedThreadPool and newFixedThreadPool both have, in different ways: under sustained overload, one creates infinite threads and the other queues infinite tasks. Neither fails loudly; both fail by exhausting resources. That's the reason most production code prefers a hand-built ThreadPoolExecutor with a bounded queue.

Other tools (work-stealing pools, the fork-join framework, virtual threads) sit on top of the same pool concept but with different optimizations. The ForkJoinPool is designed for recursive task decomposition and uses per-thread queues with work-stealing to keep cores busy. Virtual threads skip the OS-thread cost entirely so the sizing question changes. Both are special cases of "a set of workers pulling from a queue", with different rules.

A small program that uses two of the pools together, the way a real service might: a fixed pool for order processing and a scheduled pool for nightly inventory reports.

Two distinct pools, each named differently (pool-1 for orders, pool-2 for the reporter). The order pool processes the 8 orders in 4-at-a-time batches because that's its worker count. The report pool fires every 700 ms on its single worker. Neither pool interferes with the other; if the report job hangs, orders keep flowing, and vice versa. That's the bulkhead pattern in action.

The exact interleaving will shift run to run, but the structure is stable. Eight orders, three or four report runs in the two-second window, all completed before shutdown. Replace either pool with a different shape and the service's behavior under load changes accordingly. That's the lever you've.