In a monolith, one part of the code can call another part in the same process. In a distributed system, that call becomes a network request to a service instance. The hard part is that the instance address can change at any time. Containers restart, autoscalers add more instances, nodes fail, deployments replace old instances, and traffic moves between regions. A hardcoded address becomes wrong very quickly.

Service discovery solves this by mapping a stable service name, such as payment-api, to the healthy addresses currently behind it. Callers ask for the name and get back an instance they can use.

The goal is simple: callers depend on names, and the platform tracks which instances are alive and ready. This chapter explains how discovery works, when the client or the platform should choose an instance, and how health checks keep bad instances out of the list.

Consider an order service that calls inventory and payment services.

In a small static deployment, configuration like this may look harmless:

The first outage exposes the problem.

The new inventory and payment instances exist, but order traffic still goes to the original addresses. Capacity was added, but the caller cannot use it.

The failure cases are predictable:

Service discovery replaces this fragile configuration with a map that the platform keeps up to date:

The caller asks for payment-api. The platform decides which healthy instance should receive the request.

A service discovery system usually does three jobs.

The registry might be a product such as Consul, the store Kubernetes uses behind the scenes, a cloud feature such as AWS Cloud Map, or a service mesh control plane that feeds data to proxies. The tool can change, but the promise stays the same: a name should lead to healthy instances.

The service registry is the source of discovery data. It tracks:

These extra details help with routing. A request for embedding-service may need a CPU pool for background indexing, while a chat request may need a nearby GPU pool because users are waiting on the response. A small test release may send only 1% of traffic to version=v3.

Registration Process

Registration can be explicit or platform-managed.

In platform-managed environments, the service process often does not register itself. The platform already knows which pods, tasks, or VMs exist, so it updates the registry from that platform state.

Registration Patterns

Self-registration: The service registers itself with the registry during startup and renews its lease while it runs.

Self-registration can work well when every service uses the same stack. The downside is that infrastructure logic leaks into application code. Every programming language needs a correct client library, and every service must handle startup, shutdown, retries, and authentication correctly.

Platform registration: The scheduler, node agent, local proxy, or controller watches running workloads and updates discovery records.

Kubernetes follows this model. Services select pods, readiness checks decide whether a pod can receive traffic, and EndpointSlices store the current set of backend pod addresses.

Platform registration is the common default today because it keeps application code simpler and makes discovery behavior consistent across languages.

There are three patterns you should recognize: DNS discovery, client-side discovery, and server-side discovery. Production systems often combine them.

DNS is the most common discovery interface because every runtime can resolve a name.

In Kubernetes, a service named payment-api in namespace production can be addressed as:

A normal Kubernetes Service resolves to a stable ClusterIP. A headless Service resolves directly to the backing pod IPs. Consul also exposes service discovery through DNS names such as:

DNS discovery is easy to adopt, but there are a few traps:

For many services, DNS plus a platform load balancer is enough. For model serving, tenant-specific routing, or other cases with many routing choices, teams usually add a smarter client, gateway, or service mesh.

In client-side discovery, the caller gets a list of healthy instances and chooses one.

The client may choose by round robin, random selection, least requests, nearby location, consistent hashing, or weights. For example, an embedding service client may prefer instances in the same availability zone to avoid extra latency and data-transfer cost. A cache client may use consistent hashing so the same key usually lands on the same backend.

Client-Side Discovery Example: Eureka

Netflix Eureka is the classic example from the early cloud microservices era. Services register with Eureka, clients fetch registry snapshots, and client libraries choose instances locally.

Eureka is still useful to understand, especially in Java/Spring systems that adopted Netflix OSS. For new systems, most teams now start with Kubernetes Services, cloud discovery, Consul, or a service mesh. Ribbon, the old Netflix client-side load balancer often mentioned with Eureka, is no longer the default choice in modern Spring Cloud stacks.

Client-side discovery gives callers more control, but it requires discipline. If the logic is wrong, every caller can repeat the same mistake.

In server-side discovery, the client sends traffic to a stable address. A load balancer, proxy, gateway, or node-level traffic layer finds the backend instances and forwards the request.

Kubernetes Services are the standard example. The client resolves a service name through cluster DNS, receives a stable virtual IP for a normal Service, and the cluster networking layer sends traffic to one ready backend.

Modern Kubernetes clusters usually use CoreDNS to resolve service names. EndpointSlices replaced the older Endpoints API because they handle large backend lists better. Traffic may flow through kube-proxy with iptables or IPVS, or through an eBPF-based traffic layer such as Cilium. Do not assume every Kubernetes cluster routes traffic in exactly the same way.

A service mesh moves much of discovery, load balancing, retries, and security into local proxies or node-level proxies. The application calls the proxy, often without being aware of it. The mesh control plane sends instance data and routing rules to those proxies.

This model is common with Envoy-based systems. It works well when teams need mutual TLS, traffic splitting, retries, automatic removal of unhealthy instances, and detailed logs, metrics, and traces across many services.

The cost is operational. A mesh adds another control plane, more moving parts, and another place to debug latency. Use it when the organization needs the control it provides. Many microservice systems run well without one.

Each discovery approach puts the routing decision in a different place. That choice affects how complex clients become, who controls routing, and how failures show up. The table below makes the tradeoffs easier to compare.

Do not choose a pattern by fashion. Choose it by ownership.

If application teams can reliably own smart clients, client-side discovery can work well. If platform teams own routing, security, and traffic policy, server-side discovery or a mesh usually fits better. If the service only needs a stable name inside the cluster, DNS-backed Services are often enough.

Discovery is dangerous when it returns bad instances. Health checks decide whether an instance should receive traffic.

A bad health check treats "process is alive" as "service is ready." Those are different states.

Beyond naming each type, the table pairs the question it answers with the action a system takes when the check fails.

For AI systems, readiness is especially important. A model-serving pod should not receive traffic just because the HTTP server has started. It may still be loading weights, warming CUDA kernels, downloading adapters, or building a tokenizer cache. Mark it ready only when it can serve the request class it advertises.

Health Endpoint Example

A readiness check reports more than a single boolean when an instance depends on several subsystems. The example shows a /ready response for a model-serving pod in both the healthy and not-ready states.

Keep liveness checks simple. A liveness check that fails because a downstream database is slow can cause a restart storm. Use readiness to remove an instance from traffic; use liveness to restart a process that cannot recover by itself.

Push and Pull Checks

Push-based checks use heartbeats or lease renewal.

Pull-based checks are performed by a registry, load balancer, orchestrator, or proxy.

Both styles are used in production. The discovery system should update quickly, but it should not switch instances in and out of traffic during every brief hiccup.

Service discovery is rarely built from scratch. Teams usually adopt a platform that already handles registration, health checks, and lookup. The right choice depends on where services run. A single Kubernetes cluster, a mix of VMs and containers, and a managed cloud environment can all lead to different tools.

Kubernetes

Kubernetes provides discovery through Services, DNS, and EndpointSlices.

Use normal Services when clients should not care which pod handles a request. Use headless Services when the client needs individual pod identities. This is common with StatefulSets, brokers, databases, and some sharded systems.

Consul

Consul is a general-purpose service discovery and service networking platform. It is common in systems with many VMs, mixed runtime stacks, or multiple schedulers where Kubernetes is not the only way services are run.

Consul supports discovery through DNS and HTTP APIs, health checks, reusable query rules, multiple datacenters, and service mesh features. It is a strong fit when services span VMs, containers, multiple clouds, and older networks.

etcd and ZooKeeper

etcd and ZooKeeper are coordination systems. They can store discovery data, leases, and watches, but application teams should avoid building a custom discovery system on top of them unless they have a clear reason.

etcd is best known as Kubernetes' backing store. ZooKeeper appears in older distributed systems and data infrastructure. Both are useful to understand because they show how registries, leases, and watch-based updates work.

Cloud Load Balancers and Cloud Service Discovery

Cloud platforms provide managed discovery and routing through services such as AWS Elastic Load Balancing, AWS Cloud Map, Google Cloud service discovery features, Azure internal load balancers, and managed Kubernetes integrations.

Managed options reduce operational work, but they come with provider-specific behavior, limits, health-check rules, and pricing. Treat those details as part of the design, not as a tiny implementation detail.

The practices below keep discovery reliable as a system grows. Most discovery problems come from a small set of causes: unstable names, careless extra data, and health checks that say a process is ready before it can actually serve traffic.

Keep Service Names Stable

Names should describe what the service does, not how it happens to be implemented today.

Implementation names age badly. Names based on what the service does usually last longer.

Use Metadata Deliberately

Metadata is extra information attached to an instance. Use it only when it supports routing or operations.

Do not let metadata become a random dumping ground. If routing depends on a key, document what it means and which values are allowed.

Separate Liveness From Readiness

Use liveness to restart broken processes. Use readiness to control traffic. This distinction prevents many avoidable incidents.

For model-serving workloads, readiness should include whether the model is loaded, required GPU or accelerator state is ready, and local files needed for that route are available.

Drain Before Shutdown

On shutdown, stop accepting new work, fail readiness, let the discovery system remove the instance, then finish requests that are already running.

This is especially important for long-running inference, streaming responses, queue consumers, and batch workers.

Design for Slightly Old Discovery Data

Discovery data is never perfectly current. Build clients and proxies as if an instance can disappear immediately after lookup.

Retries are not a substitute for correctness. Retrying a payment or job submission that is not safe to repeat can create duplicate side effects.

Watch DNS and Connection Caching

DNS TTL does not guarantee fast traffic movement if clients hold long-lived connections. HTTP keep-alive, HTTP/2, gRPC, database pools, and SDK-level caches can keep using old instances.

For services that need fast failover, tune connection lifetimes, idle timeouts, resolver behavior, and load-balancer health thresholds together.

Avoid Thundering Herds

When a registry recovers, thousands of clients may refresh at once.

Add jitter to refresh intervals, retries, and reconnect loops.

Service discovery maps stable service names to healthy, current instances, because static IP configuration fails when services scale, fail, restart, or move. A registry stores service names, instance addresses, health state, and routing details. DNS-based discovery is simple and widely supported, though caching and long-lived connections need attention.

Client-side discovery gives callers more control but pushes complexity into every client. Server-side discovery keeps clients simple and lets infrastructure own routing.

Modern Kubernetes discovery is built around Services, CoreDNS, and EndpointSlices. Health checks must separate liveness from readiness, especially for slow-starting AI workloads such as model servers.

Good discovery design assumes data can be slightly old, instances can fail after lookup, shutdown should be graceful, retries need jitter, and ownership must be clear between application and platform teams. Discovery answers where a service is. The next question is who should be allowed to call it and through which entry point.