Placing an order looks like one action to a user. Behind the scenes, it may touch several services: orders, inventory, payments, shipping, and notifications. Each service may have its own database.

If all of that data lived in one database, life would be simpler. The database could commit everything together or roll everything back together. Distributed systems remove that single point of control.

Once one business operation spans services that commit independently, the hard question is: what happens when some steps succeed and a later step fails? If payment succeeds but the order is never confirmed, the customer may be charged for something that never ships.

This chapter explains why local database transactions do not naturally extend across services, why distributed failures are hard to read, and what a good solution must handle.

In a single database, the transaction manager can see all the data involved.

The database provides ACID guarantees:

This works because one system owns the whole decision. The database can lock rows, track uncommitted changes, check rules, and recover from its own log.

Now split the same operation across services.

Each service can make a correct local decision and still leave the full workflow in a bad state.

The order database cannot roll back the inventory database. The inventory service cannot know the payment result unless someone tells it. The payment service may even be an external provider.

Once a service commits a local transaction, undoing it is no longer a database rollback. It becomes a new business action: cancel the order, release the reservation, void the authorization, or issue a refund.

Putting all data in one database can be the right answer for many systems, especially early in a product's life. A modular monolith with strong local transactions is often simpler and safer than splitting into microservices too early.

But large systems sometimes split ownership for good reasons:

The trade-off is real. When services own their own data, some consistency work moves out of the database and into the application design.

Distributed failures are often partial. One service may commit, another may time out, and a third may still be working.

Partial Commit

A four-step order workflow commits two steps before payment fails. The workflow is now stuck halfway through.

The first two services did exactly what they were asked to do. The overall business action is still incomplete.

Network Partition

When the order service and payment service sit in separate data centers, a broken network link can leave each side unsure what the other side is doing.

When services cannot communicate, the caller does not know whether the remote service is down, slow, isolated, or still doing the work.

Timeout Uncertainty

A timeout is not an answer. It only means the caller stopped waiting.

Breaking that uncertainty into concrete cases shows why a blind retry can be dangerous:

This is why production payment and ordering systems use idempotency keys, saved request records, unique constraints, and reconciliation jobs. Retrying is necessary. Retrying without protection is dangerous.

Service Crash

Here the payment service crashes after inserting a row but before committing. From the caller's point of view, it just goes silent.

The payment was not committed, but the caller does not automatically know that. A crash after commit but before the response would look almost the same to the caller.

Local transactions usually give one application a strong view of one database. Distributed systems force you to choose where strong consistency is worth the cost.

Strong Consistency

Strong consistency means reads return the latest committed value, based on the system's rules. In practice, this usually requires coordination before a write is confirmed or before a read is served.

The cost is extra latency and lower availability when the required participants cannot communicate.

Eventual Consistency

Eventual consistency means replicas or services may disagree for a while, but they eventually catch up if updates stop and messages are delivered.

The benefit is lower latency and better availability. The cost is that application code must handle temporary intermediate states.

The CAP Connection

During a network partition, a distributed system cannot always provide both a consistent answer and an available answer.

This is the practical lesson from CAP. Network partitions are not rare surprises. They are part of running distributed systems. Your design must decide what happens when services cannot coordinate.

Inconsistent state does not stay theoretical. It becomes support tickets, accounting gaps, unavailable inventory, and manual cleanup.

Double Charge

Scenario: payment succeeds, but the response is lost.

Unconfirmed Order

Scenario: payment succeeds, but order confirmation fails.

Orphaned Reservation

Scenario: inventory is reserved, but payment fails.

Conflicting Decisions

Scenario: two services make local decisions using stale information.

Distributed transaction systems are built from parts that communicate over unreliable networks and fail independently.

No Shared Memory

Services cannot share local variables, process memory, or database locks. They coordinate by sending messages.

Messages can be delayed, lost, duplicated, reordered, or delivered after the caller has already given up.

No Perfect Failure Detector

If a service does not respond, you cannot tell from the outside what happened. It may have crashed. It may be paused. It may have lost network connectivity. It may have completed the work and lost the response.

Timeouts are useful engineering tools. They are not proof that the work failed.

No Reliable Global Clock

Each machine has its own clock, and clocks drift.

Timestamps are useful for logs, ordering events inside one system, and resolving conflicts. They are not a perfect source of truth for ordering transactions across services.

Agreement Needs Durable State

If a coordinator asks three participants to commit, every participant must be able to recover its local transaction state after a crash. That means durable logs, transaction IDs, retries, and careful handling of duplicate messages.

This is why distributed transactions are more than "call three services and roll back if something fails." The hard part is recovery after failures at every point in the workflow.

A practical design must answer these questions:

No pattern gives perfect all-or-nothing behavior, isolation, availability, low latency, and simple operations across every service. Good systems choose the right trade-off for each workflow.

Several patterns address distributed transactions, each with different trade-offs.

The table pairs each pattern with where it fits and what it costs:

Distributed transactions are hard because the system loses the thing local transactions depend on: one database with control over all the data. In a distributed workflow, each service commits its own local transaction, so later failures cannot be erased by one database rollback.

Timeouts do not reveal what actually happened. Retries can create duplicates unless operations are idempotent. Temporary states must be valid and observable. Recovery depends on saved state, not error handling alone.

Instead of pretending distributed workflows behave like one database transaction, design clear coordination, compensation, retry, and reconciliation paths. The goal is not to make failures disappear. The goal is to guide the system back to a correct business outcome.