Beyond CRUD: Building Reliable Software Systems · Part 9
New to this series? Start with Part 1
Saga Pattern Explained: Managing Distributed Transactions Across Microservices
“A transaction can roll back one database. A Saga coordinates many databases that have never even met.”
Imagine you’re building an online marketplace. A customer clicks Place Order, expecting what feels like a single operation. Behind the scenes, however, that simple button sets off a chain of events involving several independent services. The Order Service creates a new order, the Inventory Service reserves the purchased items, the Payment Service charges the customer’s card, the Shipping Service schedules delivery, and the Notification Service sends a confirmation email.
From the customer’s perspective, it’s one transaction. From your system’s perspective, it’s anything but. Each service owns its own database, each commits its own transaction independently, none of them has direct control over the others, and no single database transaction can span all of them.
Now imagine the following sequence: the Order Service successfully creates the order, the Inventory Service reserves the last laptop in stock, and the Payment Service charges the customer’s credit card. Then, just as the Shipping Service begins preparing the shipment, it discovers that delivery isn’t available to the customer’s location. The order already exists, the customer’s card has already been charged, and inventory has already been reserved, but the shipment can never be created. Unlike a traditional database transaction, there is no single ROLLBACK command capable of undoing work performed across multiple independent databases.
This is one of the defining challenges of distributed systems. As applications evolve from monoliths into microservices, business processes increasingly span services that are independently deployed, independently scaled, and independently owned. While this architecture offers tremendous flexibility, it also introduces a difficult question:
How do you maintain business consistency when a workflow spans multiple services and one of them fails halfway through?
The answer is not a larger transaction. It’s a different way of thinking. Instead of trying to make every service commit simultaneously, modern distributed systems break long-running business processes into a sequence of smaller local transactions. If every step succeeds, the workflow completes successfully. If a step fails, previously completed work is undone using carefully designed compensating actions rather than database rollbacks.
This approach is known as the Saga Pattern. Like the Outbox Pattern, the Saga Pattern embraces the reality that failures are inevitable. Rather than pretending distributed transactions behave like local database transactions, it provides a practical way to recover when things don’t go according to plan. Before we explore how Sagas work, it’s important to understand why traditional transactions stop working once your application crosses service boundaries.
Why Database Transactions Don’t Scale Across Microservices
Earlier in this series, we explored database transactions and learned how they guarantee that multiple operations either succeed together or fail together. If an online banking application deducts money from one account and credits another within the same database, a transaction ensures that both operations are treated as a single unit of work. If anything fails before the transaction commits, every change is rolled back automatically, leaving the database in a consistent state.
That model works beautifully when everything happens inside one database, but microservices change the picture completely. Imagine an order workflow involving four independent services.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Customer
│
▼
Order Service
│
▼
Inventory Service
│
▼
Payment Service
│
▼
Shipping Service
Each service owns its own database. The Order Service cannot directly roll back changes made by the Payment Service, the Payment Service cannot undo inventory reservations, and the Shipping Service has no authority over the Order database. Every service commits its own transaction independently. This independence is one of the greatest strengths of microservices, and it’s also one of their biggest challenges. Suppose the workflow progresses like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
Create Order ✅
↓
Reserve Inventory ✅
↓
Charge Payment ✅
↓
Create Shipment ❌
At this point, three services have already committed their work, and nothing can simply “roll back.” Unlike a single database transaction, there is no global undo button. This challenge is often referred to as a distributed transaction, and solving it using traditional database techniques quickly becomes impractical.
Years ago, distributed systems experimented with approaches such as Two-Phase Commit (2PC), where every participating system agreed to either commit or roll back together. While theoretically elegant, 2PC introduced significant coordination overhead, reduced availability, increased latency, and created situations where entire systems could become blocked waiting for slow or unavailable participants. Modern cloud-native architectures generally avoid this approach. Instead of attempting to make every service commit simultaneously, they accept that each service commits independently and focus on coordinating the overall business process.
That’s exactly what the Saga Pattern does. Instead of one large transaction, a Saga is a sequence of smaller transactions linked together by business logic. If every step succeeds, the Saga completes successfully; if one step fails, previously completed steps are compensated through additional business actions designed to reverse their effects. That distinction is subtle but incredibly important: a Saga doesn’t roll back database transactions, it performs new transactions whose purpose is to restore the system to a valid business state. Understanding that difference is the key to understanding everything else about the Saga Pattern.
Understanding Compensating Transactions
One of the biggest misconceptions developers have when they first encounter the Saga Pattern is assuming it somehow provides a distributed version of ROLLBACK. It doesn’t, and in fact, that’s one of the defining characteristics of a Saga. Once a service commits its local transaction, that transaction is permanent. The database has already saved the changes, and there is no mechanism for another service to rewind history. Instead of rolling back completed work, a Saga performs compensating transactions. A compensating transaction is simply another business operation whose purpose is to undo the effects of a previous one.
Suppose an order has already been created, inventory has been reserved, and payment has been successfully processed. If shipping later fails because the customer’s address falls outside the delivery area, the system cannot ask every database to roll back, because those transactions finished long ago. Instead, the application performs a series of new operations: the payment service issues a refund, the inventory service releases the reserved stock, and the order service changes the order status from Pending to Cancelled.
Notice something important: nothing has been deleted and nothing has been rolled back. The system simply performs additional work that restores the business to a valid state.
Conceptually, the workflow now looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Create Order ✅
↓
Reserve Inventory ✅
↓
Charge Payment ✅
↓
Create Shipment ❌
↓
Refund Payment
↓
Release Inventory
↓
Cancel Order
This is the heart of the Saga Pattern. Rather than pretending failures never happened, the system accepts them and responds with carefully designed business actions. This approach is much more realistic because, in distributed systems, failures aren’t exceptional, they’re inevitable.
A Banking Example
Imagine a customer applies for a personal loan. Several independent services participate in the approval process: the Loan Service creates the application, the Credit Service performs a credit check, the Risk Service evaluates affordability, and the Notification Service informs the customer of the decision. Everything proceeds normally until the Risk Service determines that the customer’s debt-to-income ratio exceeds the organization’s lending policy. At this point, the application cannot continue. If this were a single database transaction, we would simply issue a rollback.
In a microservice architecture, however, the Loan Service has already committed the new application and the Credit Service has already stored the completed credit assessment. Neither service can magically erase its work because another service encountered a problem. Instead, the Saga performs compensating actions: the Loan Service marks the application as withdrawn, the Credit Service archives its assessment, and the Notification Service informs the customer that the application could not proceed. Each action is itself a normal transaction, and collectively they restore the overall business process to a consistent state.
Rollback vs Compensation
Although these ideas sound similar, they’re fundamentally different. A database rollback behaves like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
BEGIN
↓
Update Balance
↓
Insert Payment
↓
Failure
↓
ROLLBACK
When the rollback occurs, the database behaves as though none of the changes ever happened. It’s as if the transaction never existed. A Saga works very differently.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Reserve Inventory
↓
Charge Payment
↓
Shipment Fails
↓
Refund Payment
↓
Release Inventory
The original payment really happened, the refund also really happened, and both become part of the permanent history of the system. That’s an important distinction, and many business domains actually require this behavior. Consider financial systems. Deleting payment records would make auditing impossible, so recording both the payment and the subsequent refund creates a complete, traceable history of what occurred. Compensation isn’t about pretending mistakes never happened, it’s about correcting them transparently.
Designing Good Compensating Actions
Writing a compensating transaction isn’t simply a matter of reversing database changes. You’re reversing business operations. For example, suppose a hotel booking system reserves a room.
The compensating action isn’t:
Delete reservation row.
It’s:
Release the room back into available inventory.
Similarly, if an airline charges a customer’s credit card, the compensation isn’t:
Delete payment record.
It’s:
Create a refund transaction.
This distinction matters because business systems are usually audited. Historical events should remain visible, and what changes is the current business state. Whenever you design a Saga, a useful question to ask is:
“If this step succeeds but a later step fails, what business action restores the system to a valid state?”
Thinking in terms of business operations rather than database updates leads to much more reliable designs.
Every Step Is Independent
Another characteristic of Sagas is that every step represents a complete, independent transaction. Suppose an order workflow consists of four services.
1
2
3
4
5
6
7
8
9
10
11
12
13
Order Service
↓
Inventory Service
↓
Payment Service
↓
Shipping Service
Each service commits its own database transaction before the next service begins, which means failures are isolated. If the Shipping Service experiences an outage, it doesn’t corrupt the Payment Service’s database; likewise, if the Payment Service fails, it doesn’t leave the Inventory Service with an unfinished SQL transaction.
Each service remains responsible for its own data, and the Saga simply coordinates how those independent pieces fit together. This separation of responsibility is one of the reasons microservice architectures scale so well. Services remain loosely coupled while still participating in larger business workflows.
Thinking in Business Processes
One of the biggest mindset shifts when working with Sagas is realizing that you’re no longer designing database transactions. You’re designing business processes. Database transactions answer questions like:
“How do I keep these SQL statements consistent?”
Sagas answer a much broader question:
“How does my business recover when part of this workflow succeeds and another part fails?”
That’s why Sagas often involve business concepts rather than technical ones: refunds, reservation cancellations, order cancellations, inventory releases, and account reversals. These aren’t database operations; they’re business operations that happen to involve databases. Once you begin thinking at that level, the Saga Pattern becomes much easier to understand because it mirrors how real businesses operate. Companies don’t erase history when something goes wrong, they perform additional actions to correct it, and software simply follows the same principle.
Two Ways to Coordinate a Saga
Now that we understand what a Saga is, another important question emerges: Who is responsible for coordinating all these steps?
Imagine once again that a customer places an order. The Order Service creates the order, the Inventory Service reserves the stock, the Payment Service charges the customer, and the Shipping Service prepares the shipment. If everything succeeds, the Saga completes; if one step fails, compensating transactions begin. Someone, or something, must decide what happens next.
There are two common ways to achieve this coordination:
- Choreography
- Orchestration
Both accomplish the same goal, but they do so in very different ways.
Choreography
Think about a group of experienced dancers performing together. Nobody stands at the front giving instructions. Each dancer knows exactly when to move because they respond to the music and to one another. Saga choreography works in much the same way. Instead of a central coordinator directing every step, each service reacts to events published by other services. Consider our order workflow.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
Customer Places Order
│
▼
Order Service
Publishes
OrderCreated
│
▼
Inventory Service
Publishes
InventoryReserved
│
▼
Payment Service
Publishes
PaymentCompleted
│
▼
Shipping Service
Publishes
ShipmentCreated
Every service only knows two things:
- The events it listens for.
- The events it publishes.
The Payment Service doesn’t know the Shipping Service exists, and the Shipping Service doesn’t know anything about the Inventory Service. Each service simply reacts whenever an event arrives. This loose coupling is one of choreography’s greatest strengths. Because services know very little about one another, adding or removing services often becomes much easier.
Suppose your business introduces a Loyalty Service that awards reward points whenever an order is completed. Nothing else needs to change. The new service simply subscribes to the PaymentCompleted event, and the rest of the system continues operating exactly as before. This flexibility makes choreography extremely attractive in event-driven architectures. However, it comes with a cost. As systems grow, understanding the overall business workflow becomes increasingly difficult. Instead of one visible process, the Saga becomes scattered across many services.
To understand why a shipment wasn’t created, you may need to inspect logs from the Order Service, Inventory Service, Payment Service, Shipping Service, and Notification Service. The workflow still exists, it’s simply distributed across the entire system.
Orchestration
Now imagine the same dancers performing with a conductor standing at the front. Instead of reacting to one another, every performer follows instructions from a single leader. This is orchestration. Rather than allowing services to coordinate themselves, a dedicated component, often called the Saga Orchestrator, controls the entire workflow. The orchestrator tells each service what to do next.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Saga Orchestrator
│
▼
Create Order
│
▼
Reserve Inventory
│
▼
Charge Payment
│
▼
Create Shipment
If every step succeeds, the orchestrator declares the Saga complete. If a failure occurs, it explicitly instructs previous services to execute their compensating transactions.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Shipment Failed
│
▼
Refund Payment
│
▼
Release Inventory
│
▼
Cancel Order
Unlike choreography, every decision is visible in one place. Need to understand the business process? Read the orchestrator. Need to change the workflow? Modify one component instead of updating several independent services. This centralized view makes orchestration particularly appealing for complex business processes involving many steps, conditional logic, or approval workflows. The trade-off, however, is tighter coupling. The orchestrator must understand every participating service, making it more aware of the overall system than any individual service would be in a choreographed Saga. Neither approach is universally better. The right choice depends on the complexity of the workflow and the level of control your application requires.
Choreography vs Orchestration
A useful way to compare them is to think about where the business logic lives. With choreography, the workflow is distributed across many services, and every service contributes a small piece of the overall process by responding to events. With orchestration, the workflow lives inside a single coordinator that explicitly directs every participant.
| Choreography | Orchestration |
|---|---|
| Event-driven | Command-driven |
| Highly decoupled | Central coordinator |
| Easy to extend | Easy to understand |
| Workflow spread across services | Workflow visible in one place |
| Can become difficult to trace | Coordinator becomes more complex |
Small event-driven systems often benefit from choreography because new consumers can easily subscribe to existing events. Larger enterprise workflows frequently choose orchestration because business rules remain easier to understand and maintain.
Real-World Examples
By now, you’ve probably encountered situations where the Saga Pattern would be useful without realizing it. An online retailer processing orders, a loan management platform approving applications, an airline booking system reserving flights, or a hotel reservation platform coordinating room availability all involve multiple independent services participating in a single business process. Consider a digital lending platform. When a borrower accepts a loan offer, several services may participate:
- The Loan Service creates the loan account.
- The Disbursement Service sends funds.
- The Accounting Service records journal entries.
- The Notification Service sends an SMS.
- The Credit Bureau Service updates the borrower’s status.
If the disbursement fails because the customer’s bank account is invalid, the system shouldn’t leave behind a partially created loan. Instead, compensating actions mark the loan as cancelled, reverse accounting entries where necessary, and notify the customer that disbursement was unsuccessful. No database rollback spans all these services, and the Saga restores business consistency through coordinated business actions.
Common Mistakes
The Saga Pattern is powerful, but it’s not a silver bullet. One common mistake is trying to treat compensating transactions as database rollbacks. They aren’t. Compensation should reverse business effects, not erase history. Another mistake is making Saga steps too large. Every local transaction should remain focused and complete quickly. Long-running database transactions reduce scalability and increase the likelihood of contention. It’s also important to remember that messaging is rarely perfect. Duplicate events, delayed delivery, and retries are normal in distributed systems. Every participating service should therefore be designed with idempotency in mind. Finally, don’t introduce a Saga simply because your application uses microservices. If a workflow only involves one service and one database, a normal database transaction is usually the simpler and better solution. Sagas solve distributed coordination problems, not ordinary CRUD operations.
Saga Pattern vs Traditional Transactions
At first glance, Sagas and database transactions appear to solve similar problems. In reality, they operate at completely different levels. A database transaction guarantees consistency within a single database; a Saga guarantees business consistency across multiple independent services. One relies on rollback, the other relies on compensation. One typically completes in milliseconds, while the other may run for several minutes, or even hours, depending on the business process. They’re not competing approaches. As you’ve seen throughout this series, they complement one another. In fact, a single Saga step usually contains its own local database transaction.
Bringing It All Together
At this point in the Beyond CRUD series, we’ve gradually built a toolkit for designing reliable backend systems. Each concept answers a different engineering question.
| Concept | Question It Answers |
|---|---|
| Idempotency | What if the same request arrives twice? |
| Race Conditions | What if multiple requests modify the same data simultaneously? |
| Database Transactions | How do I keep multiple database operations atomic? |
| Isolation Levels | What should concurrent transactions be allowed to see? |
| Distributed Locks | How do multiple application instances coordinate shared work? |
| Outbox Pattern | How do I reliably publish events after committing data? |
| Saga Pattern | How do multiple services complete one business process reliably? |
Notice how every concept builds upon the previous one. None replaces the others, and reliable distributed systems emerge when these patterns work together.
Final Thoughts
Building software inside a single database is relatively straightforward. Building software that spans dozens of independent services is something else entirely. The challenge isn’t simply writing correct code, it’s ensuring that business processes continue making sense even when networks fail, services restart, or one step succeeds while another doesn’t.
The Saga Pattern embraces these realities rather than fighting them. Instead of relying on one enormous transaction that spans every service, it coordinates many smaller transactions while providing a structured way to recover when failures occur. That mindset has become one of the defining characteristics of modern cloud-native architecture. As your systems continue growing, you’ll discover that reliable software isn’t built by eliminating failures, it’s built by designing systems that expect failures and know exactly how to recover from them.
What’s Next?
So far, we’ve focused primarily on making writes reliable, but as applications grow, another challenge begins to emerge. Reading data efficiently often requires very different models from writing it. Should the same model be responsible for both, or should we optimize reads and writes independently?
In the next article, we’ll explore CQRS Explained: Separating Reads and Writes for Scalable Systems, a pattern that allows applications to scale, simplify complex queries, and build richer user experiences without overloading their write models.