Scaling starts with a bottleneck.

Maybe the CPU is maxed out. Maybe memory is full. Maybe disk, network, database locks, GPU memory, or a shared service is the real limit.

Once you know what is limiting the system, there are two basic ways to add capacity.

Vertical scaling makes an existing machine or runtime bigger: more CPU, more memory, faster storage, or a larger instance type.

Horizontal scaling adds more machines, containers, processes, or nodes and spreads work across them.

Neither option is always better. The right choice depends on where the load is, how much state the component owns, how quickly demand changes, and what kind of failure the business can tolerate.

Vertical scaling is simpler but has a hard upper limit. Horizontal scaling can go much further, but it adds coordination, data splitting, and more moving parts.

Most production systems use both.

This chapter covers vertical and horizontal scaling, their trade-offs, and how systems use both.

Vertical scaling, or scaling up, means giving an existing unit more resources.

That unit might be a physical server, cloud VM, database instance, Kubernetes pod, cache node, or GPU worker.

The idea is the same in every case: keep the system shape mostly unchanged, but give one unit more room to work.

Common examples include moving a database from 8 vCPU and 32 GB RAM to 32 vCPU and 256 GB RAM, or adding memory so the hot data fits in cache.

A write-heavy database might benefit from faster NVMe storage or higher disk I/O. A Kubernetes workload may need higher CPU and memory limits.

An inference service might move to a larger GPU instance because the model no longer fits comfortably in memory.

Vertical scaling is often the first practical move because it avoids redesigning the application.

Vertical scaling is not a beginner-only strategy. Many serious production systems run for years on larger databases or search clusters because the simpler design is more reliable than spreading things out too early.

Horizontal scaling, or scaling out, means adding more units and spreading work across them.

For an application tier, this usually means running more instances behind a load balancer. For workers, it means adding more consumers to a queue.

For storage, it may mean replicas, partitions, or shards. For AI systems, it may mean more model-serving replicas, more embedding workers, or more vector database nodes.

Horizontal scaling works best when each unit can handle work independently.

A stateless HTTP service is a clean example. Any instance can serve any request, and shared state lives outside the instance in a database, cache, object store, or token.

Add more instances and the load balancer has more places to send traffic.

Stateful systems are harder. If data is tied to a specific node, the system needs routing, replication, agreement between nodes, rebalancing, and recovery logic.

Horizontal scaling can add a lot of capacity, but that capacity is not free.

If the database primary can handle 20,000 writes per second, adding 50 more web servers does not raise that limit. It may only make the database fail faster.

Different components scale differently. A good design does not apply one rule everywhere.

Stateless API servers and background workers usually scale out cleanly, as long as sessions, files, and changing state live outside the instance. Workers should also be safe to retry, because failed jobs often run again.

The number of waiting jobs is a useful scaling signal for worker systems. If the queue keeps growing, you may need more workers.

Relational databases are a different story. Read-only copies can help with read traffic, but write traffic usually needs harder changes such as partitioning or sharding.

Caches sit in a similar middle ground. Copying or splitting cache data can help, but one very popular key can still overload one node.

Search indexes and vector databases can both scale horizontally, but there are costs. Shards add capacity, but they also make rebalancing and ranking harder. A single query may also need to ask many shards before it can return an answer.

Vector databases gain storage and more work per second from extra nodes, but recall, indexing cost, and data placement become important design choices.

Model inference often scales by running more copies of the model. Large models may also need bigger machines, batching, model compression, or splitting one model across multiple GPUs.

This is why "just scale horizontally" is incomplete advice. The web tier may scale out easily while the database, embedding pipeline, or GPU inference tier becomes the real limit.

Start with measurement. Look at resource use, what is maxed out, p95 and p99 response time, queue length, error rates, database waits, and cost per request. Scaling before measuring often hides the real issue.

Choose Vertical Scaling When

Vertical scaling is usually a good first move when one machine is the clear bottleneck. For example, CPU, memory, disk I/O, network bandwidth, or GPU memory may be maxed out on that machine.

It also fits stateful workloads. Databases, caches, and search nodes often benefit from larger machines before they benefit from being split across many machines.

When the traffic forecast does not justify sharding, multi-node coordination, or more things for the team to run, vertical scaling keeps things simple.

There are two more situations where scaling up wins. If the hot data almost fits, more RAM can keep it in memory and avoid expensive disk reads.

If the code is difficult to distribute, as with legacy systems, monoliths, or tightly coupled services, a larger machine may be the only safe option until the architecture catches up.

For example, a PostgreSQL primary may be slow because the hot indexes no longer fit in memory. Moving to a larger instance can be the right next step while you optimize queries and plan longer-term partitioning.

Choose Horizontal Scaling When

Horizontal scaling is usually the better direction when the service must survive machine failures. One node should not take down the whole service.

It is also a natural fit when traffic comes in bursts, because more instances can absorb spikes. This works especially well for stateless services and queue workers.

Workloads with independent requests, jobs, tenants, files, embeddings, or messages can often be split across workers without much coordination.

Scaling out also becomes necessary when one machine is near its practical limit and the next larger machine is unavailable, too expensive, or still not enough.

Serving users from nearby regions is another driver, especially when users or data are spread around the world and delay or data-location rules require local serving.

For example, an API service is CPU-bound during peak traffic, stores no local session state, and depends on a database that still has extra room. Adding more API instances behind a load balancer is the natural move.

Most production systems combine both.

A horizontally scaled service still needs each node to be sized well. A database shard may still need to be a large machine.

A model-serving cluster may run several replicas, each on a GPU instance large enough to hold the model and serve batches efficiently.

A common combination is a bigger database paired with more application servers, where the stateless tier scales out while the database stays on a larger primary.

Read-heavy systems often add read-only copies on top of a larger primary database. Reads move to the copies while writes stay on the stronger primary.

Sharded databases follow a similar mixed pattern: split data across shards, but size each shard for its hot data and write load.

The same logic applies to background work and AI workloads. Queue workers scale out to do more work per second, and each worker instance grows when individual jobs need more CPU, memory, or GPU capacity.

Inference systems add more model-serving instances to survive failures and do more work per second. Then each instance is tuned for GPU memory, batch size, and delay targets, often with batching.

Automatic scaling does not remove the need for design. Horizontal scaling can add instances when metrics rise, and vertical scaling can adjust CPU and memory settings or recommend larger sizes.

Both still depend on good metrics, startup time, shared-dependency capacity, and safe deploy behavior.

Use this order when deciding how to scale:

The best scaling strategy is the one that increases capacity without creating a system the team cannot safely run.

Vertical scaling gives you simplicity and keeps related data and compute close together. Horizontal scaling helps you handle bursts and survive failures. Mature systems use both, but they apply each one where it matches the actual workload.

Scaling starts from a measured bottleneck, whether that is CPU, memory, disk I/O, network, GPU memory, or an overloaded shared dependency. Vertical scaling makes an existing machine or runtime bigger. Horizontal scaling adds more machines, containers, or nodes.

Vertical scaling is simple and often the fastest safe move for stateful components, but a single machine has a ceiling and can still be a single point of failure.

Horizontal scaling helps with bursts and failures, but it usually requires stateless services or carefully split state. It also requires protecting shared dependencies with connection limits, queues, rate limits, backpressure, or load shedding.

The practical order is to find the bottleneck with metrics, remove obvious waste first, scale vertically for simple extra room, and scale horizontally when failure handling or total capacity demands it. The bottleneck moves after every meaningful change, so re-test under realistic load.

Mature systems use both and apply each where it matches the workload.