Where do these locks actually live?

Unlike database transactions, distributed locks cannot rely on the memory of a single application server. Remember, our application might be running on five, ten, or even hundreds of machines, and every server must consult the same source of truth before deciding whether it may perform a particular operation.

Over the years, several technologies have emerged to solve this coordination problem. Although they all provide distributed locking capabilities, they were designed with slightly different goals in mind. Let’s look at the most common ones.

Redis

For most web applications, Redis is by far the most popular choice. Originally designed as an in-memory data store, Redis is incredibly fast, making it an excellent candidate for lightweight coordination tasks.

Acquiring a lock in Redis is surprisingly straightforward. A server attempts to create a key using an atomic command that says, in effect:

“Create this key only if it doesn’t already exist.”

If Redis successfully creates the key, the server owns the lock; if the key already exists, another server is already performing the work. Because Redis executes this operation atomically, two servers can never successfully create the same lock at the same time.

A simplified example looks like this:

SET invoice-generation

Server-A

NX

EX 30

The command says:

• Create the key only if it doesn’t already exist (NX).
• Automatically expire it after thirty seconds (EX 30).

In one atomic operation, Redis both acquires the lock and ensures it won’t remain forever if the server crashes. For many applications, this is all that’s needed.

Why Redis Is So Popular

Redis has become the default choice for distributed locks because it satisfies three important requirements. First, it’s extremely fast: since Redis stores data in memory rather than on disk, lock acquisition usually takes only a few milliseconds. Second, many applications already use Redis for caching, sessions, queues, or rate limiting, so adding distributed locking often requires little additional infrastructure. Finally, Redis has mature client libraries for virtually every programming language, with Laravel, Django, Spring Boot, .NET, Node.js, and Go all providing excellent Redis support.

For the vast majority of business applications, Redis offers an excellent balance between simplicity, performance, and reliability.

The Challenge with a Single Redis Instance

Suppose your entire application depends on one Redis server. Everything works perfectly, then Redis crashes. Suddenly, no application server can acquire new locks, and even worse, if Redis loses its in-memory state during a restart, locks may disappear unexpectedly.

This introduces a new challenge: the coordinator itself has become a single point of failure. For many applications, this risk is acceptable. For others, particularly financial systems or globally distributed services, it isn’t. This challenge led to one of the most discussed topics in distributed systems: the Redlock algorithm.

The Redlock Algorithm

Redlock was proposed by Redis creator Salvatore Sanfilippo as a way to make Redis-based distributed locks more resilient. Instead of relying on one Redis server, Redlock uses multiple independent Redis instances.

Imagine five Redis servers.

Redis 1

Redis 2

Redis 3

Redis 4

Redis 5

When acquiring a lock, the application attempts to obtain it from all five servers, and the lock is considered successful only if a majority agree.

For example:

Acquire Lock

↓

Redis 1 ✅

Redis 2 ✅

Redis 3 ✅

Redis 4 ❌

Redis 5 ❌

Three out of five succeeded, so the application proceeds. If only two servers grant the lock, the operation fails because a majority wasn’t reached. This approach significantly reduces the impact of individual Redis failures.

However, Redlock is also one of the most debated algorithms in distributed systems. Some engineers argue that it’s sufficient for many practical systems, while others, including Martin Kleppmann, have published detailed critiques explaining situations where Redlock may not provide the guarantees developers expect. The important lesson isn’t that Redlock is good or bad; it’s that distributed systems involve trade-offs, and understanding those trade-offs matters more than memorizing algorithms.

ZooKeeper

Long before Redis became popular for distributed locking, many large distributed systems relied on Apache ZooKeeper. ZooKeeper was designed specifically for coordination. Rather than functioning as a cache, it provides services such as:

• Distributed locks
• Leader election
• Configuration management
• Service discovery

Think of ZooKeeper as a highly reliable coordinator for distributed applications. Its primary goal isn’t speed, it’s correctness. Large systems such as Apache Kafka, Hadoop, and HBase have historically relied on ZooKeeper to coordinate clusters of machines. If your application requires complex distributed coordination rather than simple locking, ZooKeeper remains an excellent choice.

etcd

If you’ve worked with Kubernetes, you’ve already encountered etcd, even if you didn’t realize it. Every Kubernetes cluster stores its configuration inside etcd. Like ZooKeeper, etcd is a distributed key-value store designed for coordination rather than caching. It provides:

• Distributed locks
• Leader election
• Configuration storage
• Consensus
• Service coordination

Unlike Redis, etcd prioritizes consistency over raw performance. Its API is also designed around long-lived leases, making lock management particularly elegant. Modern cloud-native applications frequently choose etcd when they already operate within Kubernetes ecosystems.

Consul

HashiCorp Consul occupies a similar space. Although many developers know Consul for service discovery, it also provides distributed locking capabilities through sessions. Organizations using Consul often rely on it for:

• Service registration
• Health checks
• Distributed configuration
• Leader election
• Distributed locks

Like ZooKeeper and etcd, Consul focuses on reliable coordination across distributed infrastructure.

Which Technology Should You Choose?

There isn’t a universal answer. Instead, the right choice depends on your application’s requirements. If you’re building a typical web application that already uses Redis, implementing distributed locks with Redis is usually the simplest and most practical solution. If you’re coordinating hundreds of services across a Kubernetes cluster, etcd may integrate more naturally with your infrastructure. If your organization already uses Consul or ZooKeeper for service coordination, leveraging those existing systems often makes more sense than introducing Redis solely for locking.

Choosing a technology isn’t just about features. It’s also about operational complexity, existing infrastructure, and the guarantees your business requires. The important thing to remember is this: distributed locks are a concept, while Redis, ZooKeeper, etcd, and Consul are simply different tools for implementing that concept. Understanding the underlying idea is far more valuable than becoming attached to a specific technology.

Do You Always Need a Distributed Lock?

Reading this article, it might be tempting to conclude that distributed locks are the solution to every concurrency problem. They’re not. In fact, many applications never need them. If your application runs on a single server, ordinary in-memory locks are often sufficient, and if your database transaction already guarantees correctness, introducing a distributed lock may simply add unnecessary complexity.

Distributed locks are powerful, but they should be introduced only when multiple independent application instances genuinely need to coordinate shared work. Like every distributed systems technique, they solve a very specific class of problems, and the best engineering decision is often knowing when not to use them.

Common Mistakes When Using Distributed Locks

Like many distributed systems concepts, distributed locks appear deceptively simple: acquire a lock, perform some work, release the lock. In practice, however, there are several subtle mistakes that can introduce bugs that are even harder to diagnose than the problem the lock was intended to solve. Understanding these pitfalls is just as important as understanding distributed locks themselves.

Assuming a Lock Lasts Forever

One of the most common mistakes is forgetting that distributed locks usually have an expiration time. Suppose a server acquires a lock with a TTL of thirty seconds, and the developer assumes the operation will always finish within that window. Months later, a new feature makes the operation take forty-five seconds. The lock expires while the first server is still working, another server acquires the same lock and begins executing the exact same task, and suddenly, duplicate work appears again.

Choosing an appropriate TTL, and renewing it for long-running tasks when necessary, is essential.

Forgetting to Release the Lock

Although expiration protects against permanent deadlocks, applications should still release locks as soon as the protected work finishes. Holding a lock longer than necessary reduces concurrency and delays other servers waiting to perform legitimate work.

A good rule is simple:

Hold the lock only for the work that genuinely requires exclusive access.

Everything else should happen outside the lock whenever possible.

Protecting Too Much Code

Developers sometimes wrap entire workflows inside a distributed lock. For example:

Acquire Lock

↓

Call External Payment API

↓

Generate PDF

↓

Upload File

↓

Send Email

↓

Release Lock

This means every other server waits while network requests, file generation, and email delivery are taking place. Often, only a small portion of the workflow actually requires exclusive access. A better approach is to keep the critical section as short as possible.

Acquire Lock

↓

Update Shared Resource

↓

Release Lock

↓

Generate PDF

↓

Send Email

The shorter the lock, the better the system scales.

Distributed Locks vs Database Locks

At first glance, distributed locks and database locks appear very similar. Both prevent concurrent operations and both coordinate access to shared resources, but they operate at completely different levels.

A database lock protects data inside the database. For example, when two transactions attempt to update the same customer record, the database can lock that row until one transaction completes. Everything happens within the database engine itself.

A distributed lock protects work performed by application servers. Instead of preventing two transactions from updating the same row, it prevents two application instances from starting the same business process. Consider generating monthly invoices: before any invoice rows even exist in the database, every server must first decide whether it should begin the job. That decision happens outside the database, and a distributed lock coordinates that decision.

An easy way to remember the difference is:

Database locks protect data. Distributed locks protect business operations.

In many systems, you’ll use both together. A distributed lock ensures only one server begins generating invoices, and database transactions then ensure every invoice is written consistently.

Distributed Locks vs Optimistic Concurrency

Another concept frequently confused with distributed locks is optimistic concurrency. Optimistic concurrency assumes conflicts are relatively rare. Instead of preventing multiple users from editing the same record, it detects whether someone else changed the data before saving.

Suppose two employees open the same customer profile, and each begins editing. The system stores a version number alongside the record. When Employee A saves, the version changes from 5 to 6. When Employee B later attempts to save, the application notices that the version has already changed. Rather than silently overwriting Employee A’s work, it rejects the update and asks Employee B to refresh the page. No locking was required; the conflict was simply detected before committing.

Distributed locks take a different approach. Instead of detecting conflicts afterward, they prevent conflicting work from starting in the first place. Neither technique is universally better: optimistic concurrency works well when conflicts are uncommon, while distributed locks work best when duplicate execution would be expensive or dangerous.

Best Practices

As with most distributed systems techniques, simplicity is your friend. If you’re considering introducing distributed locks into your application, the following guidelines will help you avoid many common problems.

Keep Critical Sections Small

Acquire the lock immediately before modifying shared resources, and release it immediately afterward. The less work performed while holding the lock, the better your system scales.

Always Use Lock Expiration

Servers crash, containers restart, and networks fail. Never assume your application will always release its lock correctly. Expiration protects the rest of the system from waiting forever.

Verify Lock Ownership

Before releasing a lock, ensure your application still owns it. Ownership checks prevent one server from accidentally deleting another server’s lock after an expiration or retry.

Design for Failure

Distributed systems should always assume that something will eventually fail. Ask yourself:

• What happens if Redis becomes unavailable?
• What happens if the server crashes?
• What happens if the network partitions?
• What happens if the lock expires unexpectedly?

Thinking through failure scenarios early often prevents painful production incidents later.

Combine Reliability Patterns

Distributed locks are rarely used in isolation. Production systems often combine multiple reliability techniques. For example:

• Idempotency prevents duplicate requests.
• Transactions guarantee atomic database updates.
• Isolation Levels provide predictable views of data.
• Distributed Locks coordinate multiple application servers.

Each technique solves a different problem. Together, they create systems that continue behaving correctly even under heavy load and unexpected failures.

Bringing It All Together

At this point in the series, we’ve explored several concepts that all contribute to building reliable backend systems. Each one addresses a different question.

Notice how these concepts complement one another. None replaces the others; reliable systems are built by combining the right tools for the right problems.

Final Thoughts

As applications grow, the biggest challenges often aren’t writing business logic. They’re coordinating work across multiple users, multiple requests, multiple transactions, and eventually multiple servers.

Distributed locks exist because modern applications are no longer confined to a single machine. They’re deployed across clusters, containers, cloud regions, and worker nodes that all need to cooperate without constantly stepping on each other’s toes.

Like transactions and isolation levels, distributed locks aren’t something you’ll use for every feature. But when you do need them, they can be the difference between a system that behaves predictably and one that quietly creates duplicate invoices, repeated payments, or inconsistent business data.

The next time you design a background job, scheduled task, or critical workflow, ask yourself one simple question:

“What happens if two servers try to do this at exactly the same time?”

If the answer is “something bad,” you’ve probably found a place where a distributed lock belongs.

What’s Next?

We’ve now covered:

• Contract Testing
• Idempotency
• Race Conditions
• Database Transactions
• Database Concurrency
• Database Isolation Levels
• Distributed Locks

So far, every concept has focused on keeping operations consistent while they’re happening. But there’s another challenge waiting. Imagine you’ve successfully updated your database inside a transaction and now need to publish an event to Kafka, RabbitMQ, or another message broker. What happens if the database commit succeeds, but publishing the event fails? Or worse, what if the event is published but the transaction rolls back?

This problem has caused countless production incidents in distributed systems. In the next article, we’ll explore The Outbox Pattern Explained: Publishing Events Without Losing Data, one of the most widely used patterns for ensuring your database and message broker stay in sync.

“Database locks protect rows. Distributed locks protect systems.”

Imagine your application has been wildly successful. What started as a simple web application running on a single server now runs across multiple machines behind a load balancer. Requests are shared between application instances, background workers process jobs independently, and scheduled tasks run on every server.

From the outside, everything looks better than ever: pages load faster, traffic scales effortlessly, and users are happy. Then one morning, your finance department calls. Every customer has received four monthly invoices.

Nothing appears wrong with the code. The invoice generation job is scheduled to run once every month, so why did it execute four times? The answer is surprisingly simple: you now have four application servers. At midnight, every server woke up, checked the scheduler, and independently decided that it was responsible for generating invoices. From each server’s perspective, everything was perfectly correct; collectively, however, they created a costly mistake.

Now imagine a different scenario. Your payment gateway sends a webhook confirming a successful payment. The webhook is delivered to one of your load-balanced servers, but a retry occurs because the payment provider doesn’t receive a response quickly enough, so another server processes the same webhook. Both servers begin updating balances, both create accounting entries, and both generate receipts.

You’ve already learned how idempotency protects against duplicate requests and how transactions ensure database operations succeed together.

But neither of those concepts answers a new question:

How do multiple application servers agree that only one of them should perform a particular piece of work?

That problem is solved by distributed locks.

As applications grow beyond a single server, distributed locks become one of the most important coordination mechanisms in modern software engineering.

Why Database Locks Are No Longer Enough

Earlier in this series, we explored database transactions and row-level locking.

Suppose two transactions attempt to update the same bank account.

Using a statement such as:

SELECT *
FROM accounts
WHERE id = 1
FOR UPDATE;

the database ensures only one transaction can modify that row at a time.

This works beautifully because both transactions are coordinating through the same database.

Now consider a different problem.

Suppose you have four application servers running the exact same code, and each server has a scheduler responsible for calculating monthly loan interest. Midnight arrives, and every server starts the same scheduled job. None of them are updating the same database row immediately. Instead, they’re deciding whether to begin an entire business process. The database has nothing to lock yet, and by the time one server begins writing data, the others have already started processing. The result is duplicated work.

Database locks are excellent at protecting individual records, but they are not designed to coordinate entire applications spread across multiple machines. This is the gap distributed locks were created to fill.

What Is a Distributed Lock?

A distributed lock is a coordination mechanism that allows multiple independent servers to agree that only one of them may perform a particular operation at a given time. Instead of protecting a single database row, a distributed lock protects an entire business activity.

Imagine a conference room with a single key. Anyone can use the room, but only the person holding the key may enter, and everyone else must wait until the key is returned. A distributed lock works in much the same way: before performing an operation, a server first attempts to acquire the lock. If the lock is available, the server proceeds; if another server already owns the lock, the operation waits, retries, or exits.

The important point is that every server asks the same central authority for permission before beginning work. That authority might be Redis, ZooKeeper, etcd, or another distributed coordination system.

A Simple Example

Imagine four application servers.

        Load Balancer
              │
   ┌──────────┼──────────┐
   │          │          │
Server A   Server B   Server C
   │          │          │
        Server D

At midnight, each server checks whether it’s time to generate invoices.

Without a distributed lock:

Server A → Generate invoices ✅

Server B → Generate invoices ✅

Server C → Generate invoices ✅

Server D → Generate invoices ✅

Four executions, four invoices, and one very unhappy finance department.

With a distributed lock:

Server A → Acquire Lock ✅

Server B → Lock Exists

Server C → Lock Exists

Server D → Lock Exists

Only Server A proceeds, and everyone else exits. The invoices are generated exactly once.

Real-World Examples

Distributed locks appear in far more places than most developers realize. Whenever multiple application instances could accidentally perform the same work, a distributed lock becomes a potential solution.

Scheduled Jobs

Suppose your loan management platform calculates accrued interest every night at midnight. Without coordination, every application server performs the calculation independently, interest may be applied multiple times, and a distributed lock ensures exactly one server performs the calculation.

Payment Processing

Imagine receiving a payment webhook from M-Pesa. Network retries cause multiple servers to receive the same notification, and without coordination, several servers may attempt to update balances simultaneously. A distributed lock allows only one server to process the payment while the others simply exit.

Inventory Management

Only one laptop remains in stock, and two servers receive purchase requests simultaneously. Each attempts to reserve the item. Although transactions protect database consistency, a distributed lock can coordinate reservation workflows across multiple application instances before they even begin modifying inventory.

Sending Emails

Marketing decides to send a promotional email to one million subscribers, and your email scheduler is deployed across five worker nodes. Without coordination, every worker starts sending the campaign and customers receive the same email five times. With a distributed lock, only one worker initiates the campaign while the others remain idle.

Report Generation

Generating annual financial reports may take several minutes. Without coordination, multiple servers might begin generating the exact same report simultaneously, wasting CPU time and increasing database load. A distributed lock ensures only one report generation process is active.

When Should You Consider a Distributed Lock?

A useful rule of thumb is to ask yourself a simple question:

What would happen if two servers performed this operation at exactly the same time?

If the answer is:

• Duplicate invoices
• Duplicate payments
• Duplicate notifications
• Duplicate accounting entries
• Duplicate interest calculations

then the operation is probably a candidate for a distributed lock.

Not every feature needs one. Most HTTP requests don’t, reading data doesn’t, and serving web pages doesn’t. Distributed locks are primarily useful for shared background work and critical business processes where duplicate execution would produce incorrect results.

Distributed Locks Are About Coordination

One misconception worth addressing early is that distributed locks replace database transactions.

They don’t. Transactions guarantee consistency inside the database; distributed locks coordinate between application servers. The two solve different problems.

In practice, many enterprise systems use both together. A server first acquires a distributed lock, then begins a database transaction. When the transaction completes successfully, the server releases the distributed lock. The lock ensures only one server performs the work, and the transaction ensures the database remains consistent while that work is being performed. Together, they provide a powerful foundation for building reliable distributed systems.

At this point, we’ve answered why distributed locks exist.

The next question is equally important:

How do multiple servers actually agree on who owns the lock?

How Distributed Locks Work

At a high level, every distributed lock follows the same basic workflow.

Before performing a critical operation, an application asks a shared coordination service for permission to proceed. If the lock is available, the application acquires it and begins its work; if another server already owns the lock, the application waits, retries later, or simply exits. Once the work has been completed, the lock is released, allowing another server to acquire it.

Although different technologies implement this process differently, the underlying idea remains remarkably simple.

Application Server

        │

Request Lock

        │

───────────────
 Lock Service
───────────────

Lock Available?

   │          │

  Yes         No

   │          │

Acquire      Wait / Retry / Exit

   │

Perform Work

   │

Release Lock

The important detail is that every application instance talks to the same lock service. Without a shared source of truth, each server would simply believe it owned the lock.

Acquiring a Lock

Imagine four servers attempting to generate monthly invoices.

Each server sends a request to Redis asking for a lock called:

invoice-generation

Redis receives the requests almost simultaneously. The first request succeeds, and Redis stores something similar to:

invoice-generation

Owner: Server A

Expires: 30 seconds

When the remaining servers ask for the same lock, Redis responds that the lock already exists. Only Server A continues. Servers B, C, and D either wait, retry after a short delay, or abandon the operation altogether.

The beauty of distributed locks lies in their simplicity: instead of every server making its own decision, they all trust a single coordinator.

Holding the Lock

Once a server has successfully acquired the lock, it proceeds with the protected operation.

This might involve:

• Generating invoices.
• Processing a payment.
• Calculating loan interest.
• Sending reminder emails.
• Synchronizing inventory.

During this period, every other server attempting the same operation sees that the lock is already owned. Rather than performing duplicate work, those servers simply back off. The lock effectively becomes a reservation saying that someone is already doing this work and others should wait.

Releasing the Lock

When the protected work completes successfully, the server releases the lock.

Acquire Lock

↓

Process Work

↓

Release Lock

Once Redis removes the lock, another server is free to acquire it if necessary. Releasing the lock is just as important as acquiring it. A lock that is never released eventually blocks every future attempt to perform that operation.

The Problem with Permanent Locks

Now consider something less pleasant.

Server A acquires the invoice-generation lock, but halfway through generating invoices, the server crashes. Perhaps the machine loses power, perhaps Kubernetes terminates the container, or perhaps someone accidentally restarts the application. The important point is that Server A never gets the opportunity to release its lock.

If the lock remained permanent, every future invoice generation attempt would fail because the system would forever believe Server A still owned the lock. This is one of the biggest differences between traditional application locks and distributed locks: distributed systems must always assume that servers can disappear without warning.

Lock Expiration (TTL)

To solve this problem, distributed locks almost always include an expiration time, often called a Time-To-Live (TTL).

Instead of storing only the lock name, the lock service stores something like:

Lock:

invoice-generation

Owner:

Server A

Expires:

30 seconds

If Server A completes successfully, it releases the lock before those thirty seconds expire. If Server A crashes, Redis automatically deletes the lock after the TTL expires, allowing another server to continue the work instead of waiting forever.

Think of it like borrowing a meeting room: rather than reserving it indefinitely, your booking automatically expires after one hour. If you forget to leave, the reservation eventually disappears and someone else can use the room. TTL prevents abandoned locks from permanently blocking the system.

Choosing the Right TTL

Choosing a lock duration isn’t as straightforward as it might seem.

Suppose generating invoices normally takes ten seconds, and a thirty-second TTL provides plenty of room for occasional delays. But what if one month the process unexpectedly takes forty-five seconds? The lock expires after thirty seconds, another server acquires it, and now both servers are generating invoices simultaneously. You’ve accidentally recreated the very problem the lock was supposed to prevent.

On the other hand, choosing an extremely long TTL isn’t ideal either. If a server crashes while holding a lock that expires after thirty minutes, every other server must wait half an hour before continuing. Finding the right TTL therefore requires understanding how long your operation normally takes, while leaving enough room for occasional delays.

Some distributed lock implementations even allow servers to periodically renew the TTL while they’re still actively working. This approach, often called a heartbeat, keeps long-running operations alive without requiring excessively long expiration times.

What Happens If Two Servers Ask at the Same Time?

One question naturally arises: what happens if two servers request the lock at exactly the same millisecond? The answer depends on the lock service.

Redis, ZooKeeper, etcd, and similar systems perform lock acquisition atomically. That means checking whether the lock exists and creating it happen as a single indivisible operation. There is never a moment when both servers successfully acquire the same lock: one request succeeds and the other fails. This atomicity is exactly what makes distributed locks reliable; without it, two servers could both believe they owned the lock, defeating the entire purpose.

Lock Ownership Matters

Imagine Server A acquires a lock. Before finishing its work, the lock expires because the TTL was too short, and Server B now acquires the same lock. Moments later, Server A finally finishes and attempts to release it. If the lock service simply deleted the lock without checking ownership, Server A would accidentally remove Server B’s lock, Server C could now acquire it, and suddenly two servers are working simultaneously again.

To prevent this, distributed lock implementations associate every lock with a unique owner identifier. When releasing a lock, the application must prove that it is still the owner; if ownership has already changed, the release request is ignored. This simple verification prevents one server from accidentally deleting another server’s lock.

Distributed Locks Aren’t Magic

It’s important to understand that distributed locks don’t eliminate failures. Servers can still crash, networks can still become partitioned, and Redis instances can still fail. Distributed locks simply provide a coordinated way for multiple application instances to make decisions despite those realities. They reduce duplicate work, improve consistency, and coordinate critical business operations, but like every distributed systems technique, they must be implemented carefully and combined with other reliability mechanisms such as transactions, retries, idempotency, and monitoring.

In the next section, we’ll explore the most common technologies used to implement distributed locks, including Redis, Redlock, ZooKeeper, etcd, and Consul, along with the strengths and weaknesses of each approach.

Now that we’ve seen the kinds of problems concurrent transactions can create, the next question is obvious:

How does a database prevent them?

The answer lies in isolation levels.

Rather than enforcing a single set of rules for every application, relational databases allow developers to choose how isolated transactions should be from one another.

This flexibility exists because different applications have different priorities.

A banking application transferring millions of shillings every day values consistency above almost everything else.

A reporting dashboard displaying website traffic may prefer speed over perfect accuracy.

Isolation levels allow the database to balance these competing requirements.

As isolation becomes stronger, transactions observe more consistent data, but the database also has to coordinate more aggressively, often reducing concurrency.

As isolation becomes weaker, transactions execute more freely, improving performance but increasing the likelihood of observing changing data.

The SQL standard defines four isolation levels.

Each one builds upon the guarantees of the previous level.

Read Uncommitted

Read Uncommitted is the weakest isolation level defined by the SQL standard.

At this level, transactions are allowed to read changes made by other transactions even if those changes haven’t yet been committed.

Returning to our banking example, imagine Alice begins transferring KES 20,000 to another account.

The database deducts the money from her balance but hasn’t yet committed the transaction.

Another transaction immediately reads Alice’s account.

Instead of seeing KES 50,000, it now sees KES 30,000, even though the transfer could still fail and be rolled back.

That second transaction has just performed a dirty read.

The advantage of Read Uncommitted is that transactions almost never wait for one another.

Because the database performs very little coordination, throughput can be extremely high.

The downside is that applications may make decisions using data that never officially existed.

For most business applications, this is unacceptable.

Imagine calculating payroll using salaries that are eventually rolled back or approving a loan based on a balance that disappears moments later.

Fortunately, very few modern relational databases actually encourage Read Uncommitted.

Many databases either discourage it entirely or internally behave more conservatively even when it’s requested.

In practice, you’ll rarely choose this isolation level for production systems.

Read Committed

Read Committed is the default isolation level in databases such as PostgreSQL, Oracle, and SQL Server.

Instead of allowing transactions to read uncommitted changes, the database only exposes data that has already been committed.

This immediately eliminates dirty reads.

Returning to Alice’s transfer, suppose another transaction checks her balance while the transfer is still running.

Instead of seeing the temporary balance of KES 30,000, it continues seeing the previously committed balance of KES 50,000 until the transfer completes.

Only after the transaction commits does the new balance become visible.

This makes Read Committed an excellent general-purpose isolation level.

Applications never observe incomplete work, while the database still allows a high degree of concurrency.

However, Read Committed doesn’t solve every problem.

Suppose your transaction reads Alice’s balance at the beginning of a report.

A few seconds later, another transaction deposits KES 100,000 into the account and commits.

If your report queries the balance again before finishing, you’ll now see a different value.

The same row has changed during your transaction.

Read Committed prevents dirty reads, but it still allows non-repeatable reads and phantom reads.

For many applications, that’s a perfectly acceptable trade-off.

Read Committed Timeline

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 150,000

COMMIT

Both values are valid.

The important difference is that Transaction A never sees incomplete or rolled-back data.

It only observes committed changes.

Repeatable Read

Suppose you’re generating an end-of-day financial report.

Your transaction calculates the total account balance across thousands of customers.

Halfway through generating the report, another transaction updates several account balances.

If your report re-reads those accounts later, the totals may no longer match the values used earlier in the report.

This is exactly the situation Repeatable Read was designed to solve.

At this isolation level, once a transaction reads a row, subsequent reads of that same row always return the same version for the lifetime of the transaction.

Even if another transaction updates the row and commits, your transaction continues working with the original version.

It’s as though your transaction receives its own private snapshot of the database.

This provides a much more consistent view of the data, making it particularly useful for reporting systems and financial calculations.

However, Repeatable Read doesn’t necessarily prevent new rows from appearing that satisfy your query conditions.

Depending on the database implementation, phantom reads may still occur, although databases like PostgreSQL use Multi-Version Concurrency Control (MVCC) to eliminate many of these anomalies without locking every row.

This is one of the reasons database behavior differs slightly across vendors.

We’ll return to that shortly.

Repeatable Read Timeline

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 50,000 ✅

COMMIT

Although the database now contains KES 150,000, Transaction A continues seeing KES 50,000 because it is working from a consistent snapshot.

Serializable

Serializable is the strongest isolation level defined by the SQL standard.

The easiest way to understand it is to imagine that every transaction runs one after another instead of simultaneously.

Internally, the database may still execute many transactions concurrently, but it guarantees that the final result is identical to some serial execution order.

Returning to our concert ticket example, suppose only one seat remains.

Customer A begins purchasing the ticket.

Customer B attempts to purchase the same seat at exactly the same time.

Under Serializable isolation, the database ensures that only one transaction succeeds.

The other transaction must either wait, retry, or fail.

The database refuses to produce a result that couldn’t happen if the transactions had executed one after another.

This provides the strongest possible consistency guarantees.

It also comes at the highest performance cost.

Serializable transactions often require additional locking, conflict detection, or transaction retries.

For systems processing large volumes of concurrent requests, this can reduce throughput significantly.

Because of this, Serializable is typically reserved for situations where correctness is absolutely critical.

Financial ledgers, securities trading systems, and certain accounting operations are common examples.

Isolation Levels at a Glance

The table makes an important pattern clear.

As isolation increases, the number of concurrency anomalies decreases.

At the same time, the amount of coordination required by the database increases.

This is why there is no universally “best” isolation level.

The appropriate choice always depends on the requirements of your application.

Database Isolation Levels in Popular Databases

One detail that often surprises developers is that not every relational database implements isolation levels in exactly the same way.

The SQL standard defines the four isolation levels, but database vendors have some flexibility in how they achieve those guarantees.

For example, PostgreSQL relies heavily on Multi-Version Concurrency Control (MVCC). Instead of locking rows aggressively, PostgreSQL keeps multiple versions of a row and allows transactions to read a consistent snapshot of the data. This approach provides excellent concurrency while maintaining strong consistency.

MySQL’s InnoDB storage engine also supports MVCC but implements certain isolation behaviors differently. In particular, its default Repeatable Read isolation level prevents many phantom reads by using a combination of snapshot reads and gap locks.

SQL Server, on the other hand, traditionally relies more heavily on locking, although it also offers snapshot-based isolation levels that can be enabled when appropriate.

As a developer, you don’t need to memorize every implementation detail.

The important lesson is this:

Always understand how your specific database implements isolation before assuming its behavior.

The SQL standard provides the vocabulary, but your database documentation explains the exact behavior.

Choosing the Right Isolation Level

After learning about all four isolation levels, it’s natural to ask:

“Which one should I actually use?”

The honest answer is:

It depends on what you’re building.

Suppose you’re developing a dashboard that displays the number of users currently online.

If the number changes while someone refreshes the page, that’s perfectly acceptable.

There’s little value in sacrificing performance just to ensure every count remains identical throughout a transaction.

Read Committed is usually more than sufficient.

Now consider a payroll system.

Calculating employee salaries requires reading thousands of records while ensuring the figures don’t change halfway through the calculation.

If one employee’s salary is updated while payroll is being processed, the final report could contain inconsistent totals.

Repeatable Read becomes a much better fit because it provides a stable snapshot throughout the transaction.

Finally, imagine a securities trading platform or a banking ledger where even a single inconsistency could have significant financial consequences.

Here, correctness is more important than throughput.

Serializable isolation is often the safest choice, even if it means transactions occasionally wait or retry.

The goal isn’t to choose the strongest isolation level.

The goal is to choose the weakest isolation level that still guarantees the correctness your application requires.

Doing so allows the database to maximize concurrency without sacrificing business integrity.

Isolation Levels and Performance

One mistake developers sometimes make is assuming higher isolation is always better.

In reality, every additional guarantee comes at a cost.

Higher isolation levels typically require the database to:

• Coordinate more transactions.
• Acquire additional locks or maintain more snapshots.
• Detect conflicts more aggressively.
• Delay or retry conflicting transactions.

As concurrency increases, these costs become more noticeable.

A high-traffic e-commerce platform processing thousands of orders every minute cannot afford unnecessary waiting if a lower isolation level already satisfies its business rules.

Likewise, a financial institution cannot sacrifice correctness simply to process a few extra transactions per second.

Finding the right balance is part of designing reliable software.

Isolation Levels vs Transactions vs Race Conditions

At this point in the series, we’ve covered three concepts that are closely related but often confused.

Let’s put them side by side.

Transactions

Transactions answer the question:

What happens if my operation fails halfway through?

They ensure a group of related database operations either all succeed together or all fail together.

Without transactions, partial updates can leave your data inconsistent.

Race Conditions

Race conditions answer a different question:

What happens if two requests modify the same data at the same time?

These problems arise because multiple users or systems interact with shared data concurrently.

The outcome often depends entirely on timing.

Transactions alone don’t eliminate race conditions.

Additional mechanisms such as locking, optimistic concurrency, or stronger isolation levels are often required.

Isolation Levels

Isolation levels answer yet another question:

While another transaction is running, what am I allowed to see?

Should your transaction observe unfinished work?

Should it continue seeing the same data even after another transaction commits?

Should it behave as though it’s the only transaction running?

Isolation levels define these rules.

Together, these three concepts form the foundation of reliable database applications.

They complement one another rather than compete.

A payment system, for example, might use:

• Idempotency to prevent duplicate payment requests.
• Transactions to ensure payment records and account balances remain synchronized.
• Read Committed or Serializable isolation to guarantee consistent reads.
• Locking to prevent concurrent modifications of the same account.

No single technique solves every reliability problem.

Reliable systems combine several techniques, each addressing a different class of failure.

Practical Advice for Backend Developers

If you’re just beginning your backend engineering journey, don’t feel pressured to master every isolation level immediately.

Instead, focus on developing the habit of asking the right questions whenever you design a feature.

For example:

• Could another user modify this data while I’m reading it?
• If the same query runs twice, should it return the same result?
• What happens if another transaction inserts new rows before mine finishes?
• Is perfect consistency necessary, or is slightly stale data acceptable?
• Would optimistic concurrency or explicit locking be a better solution?

Thinking through these questions early in the design process often prevents bugs that are incredibly difficult to diagnose later in production.

Many concurrency issues aren’t caused by writing incorrect code.

They’re caused by making incorrect assumptions about how multiple users interact with the same data simultaneously.

Final Thoughts

Database isolation levels are often presented as a collection of definitions that developers are expected to memorize.

In reality, they’re much simpler than they first appear.

They’re simply different answers to one question:

How much should one transaction be allowed to observe while another transaction is still working?

Lower isolation levels prioritize concurrency, allowing more users to interact with the database simultaneously.

Higher isolation levels prioritize consistency, ensuring every transaction sees a predictable view of the data.

Neither approach is universally correct.

The right choice depends entirely on the problem you’re solving.

As your applications grow, understanding isolation levels becomes increasingly important because concurrency is no longer the exception—it’s the norm.

Every online store, banking application, inventory system, booking platform, and loan management system eventually reaches a point where multiple transactions compete for the same data.

The developers who understand isolation levels don’t simply build applications that work.

They build applications that continue working correctly under real-world load.

What’s Next?

In this series we’ve explored:

• Idempotency
• Race Conditions
• Database Transactions
• Database Isolation Levels

We’ve learned how to protect our systems from duplicate requests, concurrent updates, partial failures, and inconsistent reads.

But one challenge still remains.

Everything we’ve discussed assumes our application is running on a single database.

What happens when your application is running on ten servers, each processing requests simultaneously?

A normal database lock isn’t always enough.

In the next article, we’ll explore Distributed Locks Explained: Coordinating Work Across Multiple Servers, where we’ll see how systems like Redis, ZooKeeper, and etcd help ensure that only one application instance performs a critical operation at a time.

“Transactions guarantee that your work completes correctly. Isolation levels determine what everyone else is allowed to see while that work is happening.”

In the previous article, we explored database transactions and learned how they ensure multiple database operations either succeed together or fail together. Transactions protect applications from partial execution, preventing situations where money is deducted from one account without being deposited into another or where inventory is reduced without successfully creating an order.

But transactions solve only part of the problem.

Modern applications rarely have just one user interacting with the database at a time. Thousands of customers may be placing orders, updating records, making payments, or querying reports simultaneously. Each of these actions runs inside its own transaction, and more often than not, several transactions are accessing the same data at exactly the same time.

This raises an important question:

What should one transaction be allowed to see while another transaction is still running?

Imagine opening your banking application to check your account balance.

At the exact same moment, your employer’s payroll system is depositing your monthly salary into the same account.

Should your transaction see the new balance immediately?

Should it continue seeing the old balance until the salary transaction finishes?

Should it wait until the payroll transaction completes before showing you anything at all?

Each answer is technically valid depending on how the database is configured.

Now imagine an online store where only one laptop remains in stock.

Customer A begins placing an order.

Before their transaction finishes, Customer B checks the product page.

Should Customer B still see one laptop available?

Should they see zero?

Should they wait until Customer A’s purchase either succeeds or fails?

Again, the answer depends on the database’s isolation level.

Isolation levels define the rules governing how concurrent transactions interact with one another. They determine whether one transaction can observe another transaction’s work before it has been completed, whether repeated reads always return the same result, and whether new rows appearing during a transaction should be visible immediately.

Although isolation levels are often introduced as an advanced database topic, they’re really about one simple idea:

How much of another transaction’s work should your transaction be allowed to see?

The answer has significant consequences for both correctness and performance.

In this article, we’ll explore why isolation levels exist, the concurrency problems they solve, the four SQL standard isolation levels, and how to choose the right one for your application.

Why Isolation Exists

To understand isolation, imagine you’re reading a book in a library.

Halfway through chapter three, someone walks over, quietly replaces several pages with new ones, and walks away.

You continue reading without realizing anything changed.

The beginning of the chapter describes one story.

The ending describes another.

Nothing makes sense.

Databases can experience a remarkably similar problem.

When multiple transactions execute simultaneously, each transaction may be reading data while another transaction is actively changing it.

Without rules governing these interactions, applications could make decisions based on incomplete information, outdated values, or data that is eventually discarded.

Isolation exists to prevent these situations.

Rather than allowing every transaction unrestricted access to every change happening in the database, the database controls what each transaction can observe and when it can observe it.

Think of it as putting walls between transactions.

Some walls are very thin.

Transactions can see almost everything happening around them.

Other walls are much thicker.

Transactions operate almost as though they’re the only users of the database.

The thicker the wall, the more isolated the transaction becomes.

The Trade-Off Between Consistency and Performance

At first glance, it might seem obvious that every database should simply use the highest possible isolation level.

After all, if stronger isolation produces more consistent data, why wouldn’t every system choose it?

The answer lies in performance.

Imagine a supermarket with only one checkout counter.

Every customer waits patiently in line.

Because only one cashier is serving customers, inventory updates happen one at a time.

Mistakes are rare.

Unfortunately, the queue becomes enormous.

Now imagine opening ten checkout counters.

Customers move much faster.

However, all ten cashiers are now updating the same inventory system simultaneously.

Keeping everything synchronized becomes much more difficult.

Databases face exactly the same challenge.

Higher isolation levels provide stronger guarantees about data consistency, but they often require additional locking, coordination, and waiting.

Lower isolation levels allow more transactions to execute concurrently, increasing throughput and reducing latency, but they also increase the likelihood that transactions observe changing data.

Isolation levels are therefore a balancing act between two competing goals:

• Consistency, ensuring every transaction sees predictable and reliable data.
• Concurrency, allowing as many users as possible to interact with the system simultaneously.

Different applications make different choices.

A banking application processing financial transfers typically prioritizes correctness over raw performance.

An analytics dashboard generating sales reports might tolerate slightly older data if it means thousands of users can run reports simultaneously without slowing the system.

Neither approach is universally correct.

The appropriate isolation level depends entirely on your business requirements.

Concurrency Anomalies: The Problems Isolation Levels Exist to Solve

Isolation levels were not invented simply to make databases more complicated.

They exist because concurrent transactions can produce behaviors that most developers would consider surprising—or even dangerous.

These unexpected behaviors are collectively known as concurrency anomalies.

Every isolation level is essentially a trade-off between preventing these anomalies and maintaining good performance.

Before discussing the isolation levels themselves, it’s important to understand the problems they are designed to solve.

The four anomalies you’ll encounter most often are:

• Dirty Reads
• Non-Repeatable Reads
• Phantom Reads
• Lost Updates

Each represents a different way concurrent transactions can interfere with one another.

Let’s begin with the simplest.

Dirty Reads

Imagine Alice has KES 50,000 in her account.

She initiates a transfer of KES 20,000 to another account.

The banking system begins processing the transaction.

The first step deducts the money from Alice’s balance.

Before the transaction finishes, another process—perhaps an ATM balance inquiry or an online banking session—checks Alice’s account.

At that moment, it sees a balance of KES 30,000.

Everything seems perfectly normal.

Then something unexpected happens.

The transfer fails because the destination account no longer exists.

The database rolls back the transaction.

Alice’s balance immediately returns to KES 50,000.

The second transaction has now made a decision based on information that never officially existed.

It observed data that was eventually discarded.

This is known as a Dirty Read.

A dirty read occurs when one transaction reads data written by another transaction before that transaction has been committed.

The easiest way to understand it is to imagine reading someone’s unfinished draft before they’ve decided whether to keep or delete it.

The version you read may never become the final version.

Making business decisions based on that draft could lead to incorrect outcomes.

Fortunately, most modern relational databases prevent dirty reads by default because they are rarely desirable in business applications.

The SQL standard still defines them because they help explain the spectrum of isolation levels.

Timeline of a Dirty Read

Transaction A                     Transaction B

BEGIN

Balance = 50,000

↓

Update Balance = 30,000

                               Read Balance = 30,000 ❌

↓

Transfer Fails

↓

ROLLBACK

Balance returns to 50,000

Transaction B has observed a value that disappeared moments later.

From the perspective of the database, that balance never officially existed.

Yet another transaction already acted as though it did.

This is precisely the type of inconsistency isolation levels are designed to prevent.

Non-Repeatable Reads

Suppose you’re building an online banking application.

A customer opens the app and views their account balance. At that moment, the database reports a balance of KES 50,000.

The customer decides to transfer KES 40,000 to another account, but before confirming the transfer, the application performs one final balance check to ensure sufficient funds are still available.

This seems like a perfectly reasonable workflow.

However, between the first balance check and the second, another transaction deposits KES 100,000 into the same account.

When the application performs the second query, the balance is no longer KES 50,000.

It’s now KES 150,000.

Nothing is technically wrong.

The second transaction committed successfully.

The balance genuinely changed.

The surprising part is that the same transaction read the same row twice and received two different answers.

This phenomenon is known as a Non-Repeatable Read.

Unlike a dirty read, the second transaction isn’t reading uncommitted data. Every value it sees has been permanently committed to the database.

The inconsistency comes from the fact that another transaction modified the row while the first transaction was still running.

Imagine reading yesterday’s newspaper while someone keeps replacing pages with today’s edition.

The information isn’t incorrect.

It’s simply inconsistent because the document changed while you were reading it.

For many applications, this isn’t a problem.

If you’re refreshing a weather dashboard or checking the number of users currently online, it’s perfectly acceptable for values to change between two queries.

However, systems that rely on a stable snapshot of data—such as financial reporting, payroll processing, or end-of-day reconciliation—often require the same query to return the same result throughout the entire transaction.

Timeline of a Non-Repeatable Read

Transaction A                     Transaction B

BEGIN

Read Balance = 50,000

                               BEGIN

                               Deposit 100,000

                               COMMIT

Read Balance = 150,000 ❌

COMMIT

Transaction A never modified the balance itself.

It simply asked the same question twice and received two different answers because another committed transaction changed the underlying data in the meantime.

Real-World Example: Updating a Customer Profile

Consider an insurance application where a customer service representative opens a customer’s profile.

The representative spends several minutes reviewing the information before approving a policy update.

Meanwhile, another employee updates the customer’s phone number and address.

When the representative finally clicks Save, the application may now be working with information that is different from what was originally displayed.

Depending on how the application handles these changes, it could accidentally overwrite newer data or make decisions using outdated information.

This isn’t a database bug.

It’s simply the natural consequence of multiple users interacting with the same record at the same time.

Applications that require users to work with a consistent view of the data often use higher isolation levels or optimistic concurrency controls to detect these situations before committing changes.

Phantom Reads

Now let’s consider a different scenario.

Instead of reading a single row twice, imagine you’re querying an entire collection of rows.

Suppose you’re generating a report showing all loan applications submitted today.

Your first query returns:

Loan Applications Submitted Today

-------------------------------

Loan #101

Loan #102

Loan #103

Total: 3

While your report is still running, another loan application is submitted and committed to the database.

A few moments later, your transaction performs the exact same query again.

This time the results look different.

Loan Applications Submitted Today

-------------------------------

Loan #101

Loan #102

Loan #103

Loan #104

Total: 4

Notice what changed.

None of the existing rows were modified.

Instead, an entirely new row appeared.

This is called a Phantom Read.

A phantom read occurs when the same query returns a different set of rows because another transaction inserted, updated, or deleted records that match the query’s search criteria.

Think of it like counting the number of people in a room.

You count 20 people.

While you’re writing the number down, someone walks into the room.

You count again.

Now there are 21 people.

Nothing about the original twenty people changed.

The difference is that a new “phantom” appeared between your two observations.

Timeline of a Phantom Read

Transaction A                     Transaction B

BEGIN

SELECT *

WHERE loan_date = TODAY

Returns 3 rows

                               BEGIN

                               INSERT Loan #104

                               COMMIT

SELECT *

WHERE loan_date = TODAY

Returns 4 rows ❌

COMMIT

Unlike a non-repeatable read, where an existing row changes, phantom reads involve the appearance or disappearance of entire rows.

Why Phantom Reads Matter

Imagine you’re calculating today’s total revenue for financial reporting.

Your reporting transaction begins at 5:00 PM and starts aggregating sales.

While it’s still processing, new sales continue being recorded.

Different parts of the report may now be working with different datasets.

The total revenue calculated on page one might not match the detailed transaction list generated on page five because new rows appeared while the report was still executing.

In reporting systems, this can produce confusing and inconsistent results.

Higher isolation levels solve this problem by ensuring the transaction sees a consistent snapshot of the data throughout its lifetime, even if other transactions continue inserting new rows.

Lost Updates

The final concurrency anomaly is perhaps the most dangerous because it silently discards valid work.

Imagine two warehouse employees looking at the same inventory record.

The system currently shows:

Laptop Stock = 10

Employee A sells one laptop.

Employee B also sells one laptop at almost exactly the same time.

Both employees read the current stock before making their update.

Each calculates the new quantity as:

10 - 1 = 9

Employee A saves.

The inventory becomes:

9

A fraction of a second later, Employee B saves.

The inventory is still:

9

One of the updates has effectively disappeared.

The correct inventory should now be 8, but because both transactions started from the same original value, one update overwrote the other.

This is known as a Lost Update.

Unlike the previous anomalies, nothing appears obviously wrong.

No errors occur.

No constraints are violated.

The database happily accepts both updates.

The problem is that one user’s work has unintentionally replaced another’s.

Lost updates are one of the primary reasons databases provide row locking, optimistic concurrency control, and stronger isolation levels.

Without these protections, applications that receive many simultaneous updates—such as inventory systems, banking platforms, or booking applications—can slowly drift away from reality without anyone noticing.

Now that we understand the problems, the next question is obvious: How do databases prevent them? That’s exactly what we’ll cover in Part 2.”

“Transactions are not just a database feature—they’re one of the fundamental building blocks of reliable software.”

Imagine you’re transferring KES 10,000 from your savings account to a friend.

From your perspective, it’s a single action. You tap “Send Money”, authenticate the transaction, and wait for confirmation. Behind the scenes, however, the banking system performs several independent operations. It verifies that you have sufficient funds, deducts the amount from your account, credits your friend’s account, records the transaction in a ledger, updates account balances, and generates a receipt.

Each of these operations is important, but together they represent a single business action: a money transfer.

Now imagine the server crashes immediately after deducting the money from your account but before crediting your friend.

The result is disastrous. Your balance has decreased, your friend never receives the money, and unless additional recovery mechanisms exist, the system is left in an inconsistent state. From a customer’s perspective, the money has simply disappeared.

The same problem appears outside banking.

An e-commerce application might create an order, reduce inventory, process a payment, generate an invoice, and send a confirmation email. If the payment succeeds but the order creation fails, the customer has paid for a product that the system doesn’t believe exists. Likewise, in a loan management system, a repayment may update the outstanding balance, post accounting entries, and generate a receipt. If only some of those updates complete, financial records quickly become unreliable.

These problems aren’t caused by bad algorithms or poor business logic. They’re caused by partial execution when only part of a larger operation succeeds.

This is precisely the problem database transactions were designed to solve.

A transaction ensures that multiple database operations behave as a single unit of work. Either every operation succeeds together, or every operation is rolled back as though nothing ever happened. There is no halfway point where your system is left in an inconsistent state.

For backend developers, understanding transactions is just as important as understanding APIs or databases themselves. They are the foundation upon which reliable financial systems, booking platforms, inventory systems, healthcare applications, and countless other business-critical systems are built.

What Is a Database Transaction?

A database transaction is a collection of one or more database operations that the database treats as a single logical operation.

Instead of thinking about individual SQL statements, think about the business process they represent.

Suppose a customer purchases the last laptop in your online store. That single purchase might require your application to:

• Create an order.
• Deduct one item from inventory.
• Reserve the shipment.
• Record the payment.
• Create an invoice.

Although these are separate SQL statements, they represent one business event. Either all of them should succeed, or none of them should.

That’s exactly what a transaction guarantees.

Without transactions, every statement executes independently. If statement number four fails, the previous three remain committed, leaving your data inconsistent.

With transactions, the database waits until you’re satisfied that every operation has completed successfully. Only then are the changes permanently saved.

Understanding Transactions Through a Simple Example

Let’s return to the banking example.

Alice wants to transfer KES 10,000 to Bob.

A simplified version of the SQL might look like this:

UPDATE accounts
SET balance = balance - 10000
WHERE account_id = 1;

UPDATE accounts
SET balance = balance + 10000
WHERE account_id = 2;

At first glance, this seems perfectly reasonable.

But imagine the database crashes immediately after the first statement executes.

Alice’s balance has already been reduced.

Bob’s balance has not increased.

The system now contains incorrect financial data.

This is why production systems rarely execute related operations independently.

Instead, they wrap them inside a transaction.

BEGIN;

UPDATE accounts
SET balance = balance - 10000
WHERE account_id = 1;

UPDATE accounts
SET balance = balance + 10000
WHERE account_id = 2;

COMMIT;

If every statement succeeds, the database executes the COMMIT, making the changes permanent.

If any statement fails before that point, the application issues a ROLLBACK, and the database restores itself to exactly the state it was in before the transaction began.

To the outside world, it appears as though the failed transfer never happened.

This “all-or-nothing” behavior is what makes transactions so valuable.

The Lifecycle of a Transaction

Every transaction follows a predictable lifecycle.

BEGIN
   │
Execute SQL Operations
   │
Everything Successful?
   │
 ┌─ Yes ─────────────┐
 │                   │
COMMIT          Changes Saved
 │
 └─ No ──────────────┐
                     │
                 ROLLBACK
                     │
          Database Restored

The process begins with BEGIN, which tells the database to temporarily hold all modifications rather than immediately committing them.

The application then performs one or more operations. These could involve inserting records, updating balances, deleting data, or modifying relationships between tables.

If every operation succeeds, the application calls COMMIT. At that point, the database permanently saves all changes.

If anything goes wrong along the way, perhaps a validation error, a database constraint violation, or even an unexpected server failure, the transaction is rolled back, discarding every change made since BEGIN.

The beauty of this model is that the database itself guarantees consistency. Developers don’t need to manually undo every failed operation because the database handles that responsibility.

The Four ACID Properties

When developers discuss transactions, you’ll almost always hear the term ACID.

Despite sounding intimidating, ACID simply describes the guarantees that modern relational databases provide when executing transactions.

Atomicity

Atomicity means that a transaction is indivisible.

Either every operation succeeds, or none of them do.

Returning to our banking example, it makes no sense for money to be deducted from one account without being added to another. The transaction must succeed completely or fail completely.

Think of flipping a light switch.

The light cannot be half on.

Similarly, a transaction cannot be half completed.

Consistency

Consistency ensures that every transaction leaves the database in a valid state.

Business rules should always remain true.

If your application enforces that inventory can never become negative, then a successful transaction should never violate that rule.

Likewise, if your accounting system requires every journal entry to balance, no committed transaction should ever leave the ledger unbalanced.

Consistency isn’t about preventing bugs in your application logic; it’s about ensuring that completed transactions respect the rules defined by your database and your business.

Isolation

Isolation becomes important when multiple users interact with the system simultaneously.

Imagine two customers attempting to purchase the last available ticket for a concert.

Without proper isolation, both requests may read the inventory before either updates it. Both believe the ticket is available, and both complete the purchase.

You’ve now sold the same seat twice.

Isolation ensures that concurrent transactions don’t interfere with one another in ways that produce inconsistent results.

In our previous article, we discussed race conditions situations where multiple requests compete to modify the same data. Isolation is one of the database mechanisms used to prevent those concurrency problems.

We’ll explore isolation levels in greater depth in the next article because they deserve an entire discussion of their own.

Durability

Durability guarantees that once a transaction has been committed, the changes are permanent.

Even if the server loses power immediately after the commit, the database ensures that committed data survives.

Modern databases achieve this through techniques such as write-ahead logging, transaction logs, and crash recovery.

For developers, the important takeaway is simple: once the database confirms a successful commit, you can trust that the data has been safely stored.

Transactions Solve Partial Failures: Not Every Problem

One misconception among newer developers is that transactions magically solve every data consistency problem.

They don’t.

Transactions protect against partial execution.

Suppose your application updates three tables and crashes after updating the second one. A transaction ensures that the database rolls everything back, preventing inconsistent data. However, transactions don’t automatically solve concurrency problems.

Imagine two users attempting to withdraw money from the same account simultaneously. Each transaction independently checks the balance before either completes. If both see the same balance and both proceed, the final result may still be incorrect depending on your isolation level. This isn’t a transaction problem. It’s a concurrency problem.

That’s why understanding race conditions and transactions together is so important. They solve different classes of reliability issues.

Common Places You’ll Use Transactions

Transactions appear almost everywhere in modern backend systems.

Payment processing is perhaps the most obvious example. Charging a customer’s card, recording the payment, updating invoices, and generating accounting entries should either all succeed or all fail together.

Inventory management systems use transactions to ensure stock counts remain accurate even when multiple customers are purchasing products simultaneously.

Booking platforms rely on transactions to reserve hotel rooms, airline seats, or event tickets without creating conflicting reservations.

Loan management systems use transactions when posting repayments, updating outstanding balances, calculating accrued interest, and recording accounting entries.

Healthcare systems use them to ensure patient records, prescriptions, billing information, and appointment schedules remain synchronized.

Any time a business operation spans multiple database changes, a transaction is usually involved.

Common Mistakes Developers Make

One of the most common mistakes is keeping transactions open for too long.

Imagine starting a transaction, calling a third-party payment API, waiting several seconds for a response, and only then committing the transaction.

During that entire period, database resources may remain locked, reducing performance for other users.

A better approach is to perform external API calls before starting the transaction whenever possible, keeping the transaction focused solely on database operations.

Another common mistake is assuming transactions automatically protect against concurrent updates. As we’ve already seen, concurrency introduces an entirely different set of challenges that require locking strategies or appropriate isolation levels.

Finally, developers sometimes forget that transactions should represent business operations not individual SQL statements. Wrapping every single query in its own transaction rarely provides meaningful benefits.

Transactions in Modern Frameworks

Fortunately, most frameworks make transactions straightforward to use.

Laravel offers the DB::transaction() helper.

Django provides transaction.atomic().

Entity Framework supports BeginTransactionAsync().

Spring Boot uses the @Transactional annotation.

Although the syntax differs, the underlying principle never changes. The framework simply tells the database when to begin the transaction, when to commit it, and when to roll it back if something goes wrong.

Understanding the concept matters far more than memorizing framework-specific syntax.

Transactions, Idempotency, and Race Conditions

If you’ve been following this series, you may have noticed that each concept addresses a different reliability challenge.

Idempotency protects against duplicate requests by ensuring that repeating the same request doesn’t produce duplicate side effects.

Race conditions occur when multiple requests compete to modify shared data simultaneously, leading to unpredictable outcomes.

Transactions ensure that a group of related database operations either all succeed together or all fail together.

Reliable backend systems typically rely on all three.

Imagine a payment API.

Idempotency prevents customers from being charged twice if they retry a request.

Transactions ensure that charging the customer, recording the payment, and updating account balances either all succeed or all fail.

Proper concurrency control ensures that two simultaneous payment requests don’t corrupt shared data.

Each concept complements the others rather than replacing them.

Final Thoughts

Transactions are one of the reasons relational databases remain so powerful. They provide developers with a reliable mechanism for preserving data integrity even when failures occur.

As systems become larger and more distributed, failures become inevitable. Servers crash, networks fail, APIs time out, and users submit requests simultaneously. Transactions don’t eliminate those realities, but they ensure your database remains consistent when they happen.

Whenever you’re implementing a feature that modifies multiple pieces of related data, pause for a moment and ask yourself:

What happens if this operation fails halfway through?

If the answer is “my system ends up in an inconsistent state,” then you’ve almost certainly found a place where a database transaction belongs.

In the next article, we’ll build on this foundation by exploring database isolation levels and why two perfectly valid transactions can still interfere with one another when they run at the same time.

The website shows:

Only 1 ticket remaining.

At exactly the same moment, someone else clicks Buy.

Both of you complete payment.

Both of you receive confirmation emails.

But there was only one ticket.

How did two people successfully purchase the same seat?

Now imagine the same thing happening in a banking application.

Two ATM withdrawals happen at almost the same time.

Both check the balance before either transaction finishes.

Both think there’s enough money.

Both approve the withdrawal.

The account ends up with a negative balance.

Neither application is necessarily “broken.”

Instead, they suffer from one of the most common problems in software engineering:

Race conditions.

Race conditions are among the hardest bugs to reproduce because they don’t happen every time. They may only appear under heavy traffic, high concurrency, or perfect timing. Everything works during testing until your application reaches production.

In this article, we’ll explore what race conditions are, why they happen, how they relate to idempotency, and the techniques developers use to prevent them.

What Is a Race Condition?

A race condition occurs when two or more operations access and modify the same piece of data at the same time, and the final result depends on the order in which they execute.

The important phrase is:

The outcome depends on timing.

That’s what makes race conditions so dangerous.

Sometimes everything works.

Sometimes everything breaks.

The exact same code can produce different results simply because two requests happened a few milliseconds apart.

A Simple Analogy

Imagine two people standing in front of a cookie jar.

The jar contains exactly one cookie.

Both people look inside.

Both see one cookie.

Both reach in.

Both believe they’ll get the cookie.

Reality says otherwise.

Only one cookie exists.

The mistake happened because both people checked the state before either updated it.

Software behaves exactly the same way.

A Real Banking Example

Suppose an account has:

Balance = $100

Two withdrawal requests arrive simultaneously.

Request A → Withdraw $80

Request B → Withdraw $50

Without synchronization:

Request A
↓

Read Balance ($100)

------------

Request B

↓

Read Balance ($100)

------------

Request A

Balance = $20

------------

Request B

Balance = $50

Depending on timing, the final balance might be:

$20

or

$50

or

-$30

None of these outcomes are guaranteed.

This is a race condition.

Why It Works During Development

Most developers test applications alone.

One request.

One browser.

One user.

Everything works perfectly.

Production is different.

Imagine:

• 5,000 users
• Hundreds of requests per second
• Multiple application servers
• Database replication
• Network latency

The probability of two requests colliding becomes much higher.

Race conditions often appear only after an application becomes successful.

Ironically, scaling your application can reveal bugs that never existed during development.

Race Conditions vs Idempotency

If you’ve read my previous article on idempotency, if not, read it first here. You might wonder whether they’re the same thing.

They’re related, but they solve different problems.

Idempotency answers:

What if the same request is sent twice?

Example:

POST /payments

↓

Retry

↓

POST /payments

The solution is an Idempotency-Key.

The same request produces the same result.

Race conditions answer:

What if different requests happen at the same time?

Example:

User A buys last ticket

↓

User B buys last ticket

These are two legitimate requests from different users.

An idempotency key won’t help because the requests are not duplicates.

Race conditions require synchronization, not deduplication.

Real-World Examples

Airline Seat Booking

Only one seat remains.

Two customers purchase it simultaneously.

Without proper locking:

• Seat sold twice
• Refund required
• Customer frustration

E-Commerce Inventory

Stock:

Laptop

Quantity = 1

Two customers purchase simultaneously.

Without protection:

Inventory becomes:

-1

Now you’ve sold a product you don’t have.

Loan Approval Systems

Imagine two loan officers reviewing the same application.

Officer A:

Approve.

Officer B:

Reject.

If both updates happen simultaneously without coordination, the final loan status depends entirely on timing.

Coupon Redemption

Promotion:

First 100 customers only

If multiple requests update the redemption count simultaneously, you might accidentally issue:

103

105

110

coupons.

How Race Conditions Happen

Most race conditions follow this pattern:

Read

↓

Modify

↓

Write

The danger lies between Read and Write.

Example:

Read Balance

↓

Calculate New Balance

↓

Update Balance

If another request changes the balance between those steps, your calculation becomes outdated.

This is known as a Lost Update problem.

Solution 1: Database Transactions

Transactions ensure multiple operations succeed or fail together.

Example:

BEGIN;

SELECT balance
FROM accounts
WHERE id = 1;

UPDATE accounts
SET balance = balance - 80
WHERE id = 1;

COMMIT;

Transactions protect data consistency.

However, they don’t automatically eliminate every race condition.

Isolation levels matter too.

Solution 2: Row-Level Locks

Many relational databases allow locking a row while it’s being updated.

Example:

SELECT *
FROM accounts
WHERE id = 1
FOR UPDATE;

Now:

Request A locks the row.

Request B must wait.

Only after Request A finishes can Request B continue.

This guarantees consistent updates.

Solution 3: Optimistic Locking

Instead of preventing conflicts, optimistic locking detects them.

Imagine a version number.

Account

Balance = 100

Version = 5

Update:

WHERE Version = 5

If another request updates the row first:

Version = 6

Your update fails.

The client retries with fresh data.

Optimistic locking works well when collisions are relatively rare.

Solution 4: Distributed Locks

What if your application runs on multiple servers?

Server A

↓

Database

↑

Server B

A normal in-memory lock won’t work because each server has its own memory.

Instead, developers use distributed locking systems like:

• Redis (Redlock)
• ZooKeeper
• etcd
• Consul

These coordinate access across multiple application instances.

Solution 5: Atomic Database Operations

Sometimes you don’t need to:

Read

↓

Calculate

↓

Write

Instead, let the database perform the update atomically.

Bad:

SELECT quantity;

quantity--;

UPDATE products;

Better:

UPDATE products

SET quantity = quantity - 1

WHERE quantity > 0;

Now the database guarantees consistency.

Detecting Race Conditions

One reason race conditions are difficult is that they rarely appear during manual testing.

Ways to uncover them include:

• Load testing with concurrent users
• Stress testing
• Running parallel integration tests
• Simulating delayed responses
• Chaos engineering
• Monitoring production logs

If a bug only appears “sometimes,” concurrency should be one of your first suspects.

Best Practices

When designing systems that handle shared data:

• Keep transactions short.
• Avoid long-running locks.
• Prefer atomic database operations where possible.
• Use optimistic locking when contention is low.
• Use pessimistic locking for critical resources.
• Test with concurrent requests, not just sequential ones.
• Understand your database’s transaction isolation levels.

Most importantly, always ask:

“What happens if two users do this at exactly the same time?”

Final Thoughts

Race conditions aren’t caused by bad developers, they’re caused by systems becoming concurrent.

As applications grow, users interact simultaneously, background jobs overlap, and services communicate in parallel. Timing becomes unpredictable.

That’s why building reliable software isn’t just about writing correct logic for one request. It’s about ensuring your logic remains correct when hundreds or thousands of requests happen together.

If idempotency protects you from duplicate requests, race condition handling protects you from competing requests.

Together, they form two of the most important building blocks for designing resilient APIs and distributed systems.

The next time you write code that reads, modifies, and writes shared data, pause for a moment and ask:

“What happens if someone else does this at the exact same time?”

If you don’t know the answer, you’ve just found your next engineering problem to solve.

You click the “Pay Now” button.

Nothing happens.

After a few seconds, you assume the request failed, so you click the button again.

And again.

A few minutes later, you discover you’ve been charged three times.

What happened?

From the user’s perspective, the payment seemed to fail. From the server’s perspective, however, it successfully processed every request it received.

This is one of the most common problems in distributed systems, and it’s exactly why idempotency exists.

Whether you’re building payment systems, booking platforms, inventory management software, or any API that changes data, retries are inevitable. Networks fail, clients time out, mobile connections drop, and users double-click buttons.

A well-designed API should survive these retries without creating duplicate side effects.

In this article, we’ll explore what idempotency is, why it matters, and how to implement it in your own APIs.

What Is Idempotency?

In simple terms, an idempotent operation can be performed multiple times without changing the final result beyond the first successful execution.

For example:

Turn on the light.
Turn on the light again.
Turn on the light again.

The light is still ON.

Nothing new happened after the first request.

The final state remains the same.

That’s idempotency.

Now compare it with this:

Deposit $100
Deposit $100
Deposit $100

Your account balance increases by $300.

This operation is not idempotent because every request changes the system.

Why Retries Happen

Many developers assume users only send one request.

Reality is different.

Requests are retried because of:

• Slow internet connections
• Gateway timeouts
• Reverse proxies
• Mobile network interruptions
• Browser refreshes
• Double-clicking buttons
• Client retry mechanisms
• Load balancers
• Microservice communication failures

Imagine this timeline:

Client -------- POST /payments --------> API

             Payment succeeds

API -------- 200 OK --------X

(Response never reaches client)

Client waits...

Client retries.

POST /payments again.

The client believes the payment failed.

The server already completed it.

Without idempotency…

The payment happens twice.

HTTP Methods and Idempotency

HTTP itself distinguishes between idempotent and non-idempotent methods.

GET

GET /users/10

Read the user.

Call it once.

Call it 100 times.

Nothing changes.

Idempotent

PUT

PUT /users/10
{
   "name": "Billy"
}

Replacing the same resource repeatedly produces the same result.

Idempotent

DELETE

DELETE /users/10

Delete the user.

Deleting an already deleted user doesn’t delete them twice.

The final state is still:

User does not exist.

Idempotent

POST

POST /orders

Create a new order.

Call it twice.

You now have two orders.

Not idempotent

This is why POST requests often require additional protection.

Why Payment APIs Use Idempotency Keys

Payment providers like Stripe popularized the use of Idempotency Keys.

The idea is simple.

The client generates a unique identifier.

Example:

Idempotency-Key:
6ab89d3b-acde-4d71-b20d-483d8d0ef091

Every retry sends the same key.

POST /payments

Idempotency-Key:
6ab89d3b-acde-4d71-b20d-483d8d0ef091

When the server receives the request:

1. Check if this key already exists.
2. If not, process the payment.
3. Save both the key and the response.
4. Return the response.

If the same request arrives again with the same key:

Instead of charging the customer again…

Return the previously stored response.

Client

POST /payments
Key: ABC123

↓

Server

Charge customer

↓

Store

ABC123 → Payment #456

↓

Return success

---

Retry

POST /payments
Key: ABC123

↓

Lookup

ABC123 exists

↓

Return Payment #456

No second charge.

Implementing Idempotency

A common workflow looks like this.

Step 1

Receive request.

POST /orders

Headers

Idempotency-Key:
XYZ987

Step 2

Search database.

SELECT *
FROM idempotency_keys
WHERE key = 'XYZ987'

Found?

Yes.

Return stored response.

Done.

Step 3

Not found?

Create the resource.

Create Order

Step 4

Store:

Key

Response

Status Code

Timestamp

Now every retry returns the same response.

Example in Node.js (Express)

app.post("/payments", async (req, res) => {
    const key = req.header("Idempotency-Key");

    const existing = await Idempotency.findOne({ key });

    if (existing) {
        return res.status(existing.status).json(existing.response);
    }

    const payment = await processPayment(req.body);

    await Idempotency.create({
        key,
        status: 201,
        response: payment,
    });

    return res.status(201).json(payment);
});

Python (FastAPI)

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/payments")
async def create_payment(request: Request):
    key = request.headers.get("Idempotency-Key")

    existing = await Idempotency.find_one(key=key)

    if existing:
        return JSONResponse(
            content=existing.response,
            status_code=existing.status,
        )

    body = await request.json()
    payment = await process_payment(body)

    await Idempotency.create(
        key=key,
        status=201,
        response=payment,
    )

    return JSONResponse(content=payment, status_code=201)

C# (ASP.NET Core)

app.MapPost("/payments", async (
    HttpRequest request,
    IdempotencyStore store,
    PaymentService payments) =>
{
    var key = request.Headers["Idempotency-Key"].ToString();

    var existing = await store.FindAsync(key);
    if (existing is not null)
    {
        return Results.Json(existing.Response, statusCode: existing.Status);
    }

    var body = await request.ReadFromJsonAsync<PaymentRequest>();
    var payment = await payments.ProcessAsync(body!);

    await store.CreateAsync(new IdempotencyRecord(key, 201, payment));

    return Results.Json(payment, statusCode: StatusCodes.Status201Created);
});

Go

func createPayment(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")

    existing, err := idempotency.FindOne(r.Context(), key)
    if err == nil && existing != nil {
        w.WriteHeader(existing.Status)
        json.NewEncoder(w).Encode(existing.Response)
        return
    }

    var body PaymentRequest
    if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    payment, err := processPayment(r.Context(), body)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    if err := idempotency.Create(r.Context(), IdempotencyRecord{
        Key:      key,
        Status:   http.StatusCreated,
        Response: payment,
    }); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusCreated)
    json.NewEncoder(w).Encode(payment)
}

Laravel

Route::post('/payments', function (Request $request) {
    $key = $request->header('Idempotency-Key');

    $existing = Idempotency::where('key', $key)->first();

    if ($existing) {
        return response()->json($existing->response, $existing->status);
    }

    $payment = processPayment($request->all());

    Idempotency::create([
        'key' => $key,
        'status' => 201,
        'response' => $payment,
    ]);

    return response()->json($payment, 201);
});

The logic is surprisingly simple.

The complexity comes from storing and managing the keys correctly.

Where Should Idempotency Keys Be Stored?

Options include:

Database

Best for most applications.

Pros:

• Persistent
• Reliable
• Easy to query

Cons:

• Slightly slower

Redis

Excellent for high-volume APIs.

Pros:

• Extremely fast
• TTL support
• Easy expiration

Many APIs automatically expire keys after 24 hours.

In-Memory

Useful only during development.

Not recommended for production.

Restarting the server loses everything.

Common Mistakes

Reusing Keys

Every logical operation should have its own unique key.

Bad:

ABC123

used today

used tomorrow

Good:

New checkout

↓

Generate new UUID

Ignoring Request Differences

Suppose the first request is:

$50

The retry is:

$500

Same key.

Different body.

The server should reject this request because the key is being reused for a different operation.

Never Expiring Keys

Keeping millions of old keys forever wastes storage.

Most APIs expire them after:

• 24 hours
• 48 hours
• 7 days

depending on business requirements.

Real-World Use Cases

Idempotency is valuable anywhere duplicate requests could have costly consequences.

Examples include:

• Payment processing
• Bank transfers
• Order creation
• Hotel reservations
• Flight bookings
• Ticket purchases
• Subscription billing
• Inventory updates
• Webhook processing
• Email sending
• Message queues

If performing the same action twice could create an incorrect outcome, idempotency is worth considering.

When You Don’t Need Idempotency

Not every endpoint needs an idempotency key.

For example:

GET /posts

No state changes.

No duplicates.

No problem.

Likewise, endpoints such as:

• Search
• Filtering
• Reading reports
• Viewing profiles

are already naturally idempotent.

Reserve idempotency mechanisms for operations where retries could create unintended side effects.

Final Thoughts

Idempotency isn’t just an implementation detail—it’s a reliability feature.

In distributed systems, retries are normal. Networks are unreliable, users click buttons more than once, and clients retry requests automatically. Instead of hoping those situations never happen, design your APIs to handle them gracefully.

By using idempotency keys, storing responses, validating retries, and choosing the right storage strategy, you can prevent duplicate orders, repeated payments, and other costly errors.

A resilient API isn’t one that never receives duplicate requests.

It’s one that produces the correct outcome even when duplicate requests inevitably arrive.

The next time you design a POST endpoint, ask yourself:

“What happens if this request is sent twice?”

If the answer is “something bad,” it’s probably time to add idempotency.

I wasn’t thinking about traffic, search engine rankings, personal branding, or becoming a better writer. I simply wanted a place to share my thoughts, document what I was learning, and build something that belonged to me.

Like many developers, I started blogging because I was learning new things every day and wanted to keep track of my progress. At the time, I had no idea how much writing would teach me—not just about technology, but about consistency, communication, patience, and personal growth.

Now, after publishing my first 72 blog posts, I’ve learned lessons that extend far beyond blogging itself. Some lessons came from successes, while others came from mistakes and moments of frustration.

In this article, I want to share the most important lessons I’ve learned from publishing my first 72 blog posts.

1. Consistency Beats Perfection

One of the biggest mistakes I made early on was trying to make every article perfect.

I would spend hours tweaking sentences, rewriting paragraphs, and overthinking every detail. Sometimes I delayed publishing because I felt an article wasn’t “good enough.”

Over time, I realized that perfection is the enemy of progress.

The articles that helped me grow weren’t necessarily my best articles. They were simply the articles I published.

Consistency creates momentum.

Publishing one article every week for a year is far more valuable than spending months trying to create a single masterpiece.

The biggest breakthrough came when I stopped chasing perfection and started focusing on showing up consistently.

2. Writing Clarifies Thinking

Many times, I thought I understood a topic until I tried to explain it in writing.

That’s when the gaps in my knowledge became obvious.

Writing forced me to organize my thoughts, simplify complex ideas, and identify areas where my understanding was incomplete.

The process taught me that learning and explaining are not the same thing.

If I couldn’t explain a concept clearly, I probably didn’t understand it as well as I thought.

This lesson made me a better learner and a better developer.

3. Nobody Reads Your First Posts—And That’s Okay

One of the hardest realities for new bloggers is that almost nobody reads your early content.

I remember publishing articles and checking analytics repeatedly, hoping to see visitors.

Most of the time, there were very few.

At first, that felt discouraging.

But eventually I realized something important:

The purpose of your first blog posts isn’t to attract thousands of readers.

The purpose is to learn how to write.

Your first posts are practice.

Every article improves your skills, your confidence, and your ability to communicate.

The audience comes later.

4. The Habit Matters More Than Motivation

Motivation is unreliable.

Some days you’ll feel inspired.

Other days you’ll have no desire to write.

If blogging depends entirely on motivation, consistency becomes impossible.

One of the most valuable lessons I learned is that habits outperform motivation.

The writers who succeed aren’t necessarily the most talented.

They’re the ones who continue writing even when they don’t feel like it.

Creating a writing habit transformed blogging from an occasional activity into a regular part of my routine.

5. Every Post Doesn’t Need to Be Revolutionary

Early in my blogging journey, I believed every article needed a unique insight or groundbreaking idea.

I was wrong.

Many of my most useful posts covered topics that had already been discussed countless times.

The difference was that I shared my own perspective and experience.

Your value doesn’t come from inventing entirely new ideas.

It comes from explaining ideas through your own lens.

There’s always someone who needs the explanation that only you can provide.

6. Writing Builds Confidence

Publishing online can feel intimidating.

You’re sharing your thoughts with strangers.

You’re exposing your ideas to criticism.

You’re putting your work in public.

At first, this can be uncomfortable.

But over time, each article builds confidence.

You become more comfortable expressing your opinions.

You stop worrying about being perfect.

You gain confidence in your ability to communicate and contribute valuable ideas.

That confidence eventually extends beyond blogging into other areas of life and work.

7. Blogging Is a Long-Term Game

One of the most important lessons I learned is that blogging rewards patience.

Results rarely happen overnight.

Traffic grows slowly.

Authority builds gradually.

Opportunities emerge unexpectedly.

Many people quit because they expect immediate results.

The reality is that blogging is similar to investing.

Small contributions made consistently over time eventually compound into significant outcomes.

The key is staying committed long enough to experience those benefits.

8. Publishing Teaches More Than Consuming

Before I started blogging, I spent most of my time consuming content.

I watched tutorials.

I read articles.

I followed courses.

While these activities were helpful, publishing taught me far more.

Creating content forces you to think deeply, research thoroughly, and communicate clearly.

It transforms you from a consumer into a creator.

That shift changed the way I learn forever.

9. Your Blog Becomes a Personal Archive

One unexpected benefit of blogging is having a permanent record of your journey.

Looking back at my older articles shows how much I’ve grown.

I can see:

• How my thinking evolved.
• What I was learning.
• Problems I was solving.
• Mistakes I made.

The blog becomes more than a collection of articles.

It becomes a timeline of personal growth.

10. Small Efforts Compound Over Time

When I published my first article, it felt insignificant.

The same was true for my second, third, and tenth article.

But by the time I reached fifty posts, those small efforts had accumulated into something meaningful.

Fifty articles represent:

• Hundreds of hours of learning.
• Hundreds of hours of writing.
• Dozens of lessons learned.
• A growing body of work.

This reminded me that success is rarely the result of a single big action.

It’s usually the result of many small actions repeated consistently over time.

11. The Real Reward Isn’t Traffic

When people think about blogging success, they often think about page views, followers, and analytics.

Those metrics matter, but they’re not the greatest reward.

The greatest reward is who you become during the process.

Blogging made me:

• A better writer.
• A better learner.
• A better communicator.
• A more disciplined creator.
• A more thoughtful developer.

Those benefits are far more valuable than any traffic number.

Looking Ahead

Reaching 72 blog posts feels like an important milestone, but it also feels like the beginning.

There’s still so much to learn.

So many topics to explore.

So many experiences to share.

The goal isn’t simply to publish more articles.

The goal is to continue learning, improving, and documenting the journey.

If the first 72 posts taught me anything, it’s that growth comes from consistency, curiosity, and the willingness to keep showing up.

Conclusion

Publishing my first 72 blog posts taught me lessons that extend far beyond writing.

It taught me patience when growth was slow.

It taught me discipline when motivation disappeared.

It taught me confidence when self-doubt appeared.

Most importantly, it taught me that meaningful progress is often the result of small actions repeated consistently over time.

If you’re thinking about starting a blog, my advice is simple:

Start now.

Don’t wait until you’re an expert.

Don’t wait until everything is perfect.

Write your first post.

Then write your second.

Then your third.

One day, you’ll look back and realize those small steps created something much bigger than you ever imagined.

I did an article about importance of writing for developers which you can check out.

You’ve been staring at the same bug for 2 hours. You’ve restarted the server. Cleared cache. Added random console.logs. Somehow… it still doesn’t work.

At some point, you stop coding and start guessing.

And that’s the real problem.

The issue isn’t the bug. It’s that nobody actually teaches debugging as a skill.

The Way Most Developers Debug

Let’s be honest. Most of us learned debugging like this:

• Sprinkle console.log everywhere
• Change random lines and hope something works
• Copy-paste error messages into Google
• Restart everything “just in case”

It sometimes works.

But it’s slow, frustrating, and unreliable.

It’s not debugging.

It’s trial and error disguised as progress.

What Debugging Actually Is

Here’s the mindset shift that changes everything:

Debugging is not about fixing code. It’s about finding where your mental model diverges from reality.

You think the system works one way.

Reality says otherwise.

Your job is to close that gap.

The Debugging Mindset

Before tools, before techniques, this is what matters most.

1. Assume Your Assumptions Are Wrong

If something doesn’t work, at least one thing you believe is false.

Your job is to find it.

2. Narrow the Problem Space

Bad debugging:

“Something is wrong with the app”

Good debugging:

“The issue happens only when this function runs after login”

3. Reproduce Before Fixing

If you can’t reliably reproduce the bug, you don’t understand it.

And if you don’t understand it, your fix is luck—not skill.

4. One Change at a Time

If you change 5 things and it works… which one fixed it?

You don’t know.

That’s how bugs come back later.

5. Understand Before You Patch

Quick fixes feel good.

Understanding the root cause makes you dangerous (in a good way).

A Repeatable Debugging Process

This is where things become practical.

Step 1: Reproduce the Bug

Make it happen consistently.

Click button → error appears  
Refresh → still happens  
Different browser → still happens

If it’s inconsistent, your first task is to find the pattern.

Step 2: Define Expected vs Actual

Write it down clearly.

Expected: API returns user data  
Actual: API returns empty array

This step alone eliminates confusion.

Step 3: Isolate the Problem

Shrink the scope.

• Comment out unrelated code
• Remove layers (UI → API → DB)
• Test pieces independently

Think of it like this:

[ UI ] → [ API ] → [ Database ]

Which layer is lying?

Step 4: Form a Hypothesis

Be explicit:

“I think the API is returning empty data because the query filter is wrong.”

Now you’re not guessing—you’re testing a theory.

Step 5: Test the Hypothesis

Use targeted tools:

• Logs
• Breakpoints
• Network inspector

Example:

console.log("User ID:", userId)

But intentional—not random.

Step 6: Fix and Verify

Fix it.

Then confirm:

• Does it work in all cases?
• Did you break something else?

Step 7: Understand the Root Cause

This is where most devs stop too early.

Don’t just fix it—explain it:

“The bug happened because the state updated asynchronously, and we read it too early.”

Now you’ve learned something reusable.

Real Example: “The API Is Broken” (But It’s Not)

Let’s walk through a real scenario.

The Bug

Frontend shows: No data available

Initial Assumption

“The API is broken.”

Step 1: Check Network Tab

You open DevTools → Network:

API returns correct data

So… not the API.

Step 2: Check State

console.log(data)

It logs:

[]

Empty array.

Step 3: Trace the Flow

useEffect(() => {
  fetchData()
}, [])

Inside fetchData:

setData(response.data)
console.log(data) // still empty

The Problem

React state updates are asynchronous.

You’re logging before state updates.

The Fix

useEffect(() => {
  fetchData()
}, [])

useEffect(() => {
  console.log(data)
}, [data])

The Lesson

The bug wasn’t in the API. It was in your mental model of how state updates work.

Tools That Actually Help

Not everything is about tools—but the right ones matter.

1. Browser DevTools (Underrated Powerhouse)

• Network tab → verify API calls
• Console → inspect runtime values
• Application tab → check storage

2. Breakpoints (Game Changer)

Instead of spamming logs:

Pause execution and inspect state live

3. Intentional Logging

Bad:

console.log("here")

Good:

console.log("User after login:", user)

4. Stack Traces

Read them.

They literally tell you:

• Where the error happened
• What triggered it

5. Rubber Duck Debugging

Explain the bug out loud.

Yes, seriously.

You’ll often solve it mid-explanation.

Problem → Hypothesis → Test → Learn → Repeat

Common Debugging Traps

Avoid these and you’ll already be ahead of most devs:

Fixing Symptoms Instead of Causes

You silence the error… but the bug is still there.

Changing Too Many Things at Once

Now you don’t know what worked.

Ignoring Error Messages

The error is literally telling you what’s wrong.

Read it.

Assuming the Bug Is “Weird”

It’s almost never weird.

It’s misunderstood.

“It Works on My Machine”

This is not a flex.

It’s a clue.

A Better Mental Model

Expectation ≠ Reality  
        ↓  
Investigate the gap

That’s debugging.

Final Thought

The best developers aren’t the ones who write perfect code. They’re the ones who can quickly understand why things break.

Debugging isn’t a side skill.

It is the job.

Heads-up: v2 uses different endpoints and JSON than classic Accounts CRUD. Always confirm API version, preview flags, and SDK support in Stripe’s documentation before shipping production code.

What is Stripe Connect?

Stripe Connect is for platforms that move money between buyers, sellers, and your platform. You connect connected accounts to your platform account so you can:

• Collect payments from customers
• Pay out to sellers or service providers
• Take application fees where appropriate

Stripe holds the payments rail; you own product and UX choices.

Two ideas at once: “style” (Standard / Express / Custom) and “shape” (v2 configurations)

The original guide compares Standard, Express, and Custom as who owns onboarding, dashboards, and compliance.

Accounts v2 adds a separate axis: what capabilities a single Account has, expressed as configurations:

You can assign one or more configurations to the same Account. That is the core promise of v2: one identity, multiple roles, less manual ID mapping.

The Standard / Express / Custom choice is still real: it describes OAuth vs Stripe-hosted onboarding vs fully custom UI. On v2 you implement those experiences while creating and updating Accounts through the v2 surface (where supported).

Standard vs Express vs Custom (unchanged product trade-offs)

On v2, you still make these product choices—but you attach merchant / customer / recipient configurations to match what each connected business must do.

Architectural overview

Fund flow is unchanged at a high level:

What changes in v2 is object modeling:

• One Account can represent both “seller” and “buyer of your SaaS” if you add merchant and customer configurations.
• APIs that took customer may accept customer_account with an Account ID—see using Accounts as customers.

Accounts v2 primitives (before code)

Stripe’s v2 Account objects differ from v1’s flat Account create. You typically send:

• identity — country, entity type, business details.
• configuration — nested blocks such as merchant, customer, recipient, each with capabilities you request (card_payments, balance / payout capabilities as named in current docs).
• defaults — currency, locales, responsibilities (fees / losses collector), etc.
• include — ask the API to return nested sections (e.g. configuration.merchant, requirements).

Responses may omit fields unless you include them—see Stripe’s note on includable response values.

API version: v2 calls often require a specific Stripe-Version header (including preview versions while the API evolves). Set this from the exact value in Stripe’s Accounts v2 docs for your integration window.

Implementation: creating an Account on v2 (HTTP-first)

Official examples are REST + JSON. Below is illustrative—copy field names, capability keys, and version from the current docs, not from this blog alone.

cURL (canonical pattern from Stripe)

Stripe documents POST /v2/core/accounts with a JSON body. Conceptually:

curl -X POST https://api.stripe.com/v2/core/accounts \
  -H "Authorization: Bearer sk_test_..." \
  -H "Stripe-Version: <version-from-stripe-docs>" \
  -H "Content-Type: application/json" \
  --data '{
    "contact_email": "seller@example.com",
    "display_name": "Example Seller",
    "identity": {
      "country": "us",
      "entity_type": "company",
      "business_details": { "registered_name": "Example Co" }
    },
    "configuration": {
      "merchant": {
        "capabilities": {
          "card_payments": { "requested": true }
        }
      },
      "customer": {
        "capabilities": {
          "automatic_indirect_tax": { "requested": true }
        }
      }
    },
    "defaults": {
      "currency": "usd",
      "responsibilities": {
        "fees_collector": "stripe",
        "losses_collector": "stripe"
      },
      "locales": ["en-US"]
    },
    "include": [
      "configuration.customer",
      "configuration.merchant",
      "identity",
      "requirements"
    ]
  }'

This shows the mental model: one create, multiple configurations, explicit includes.

Python (generic HTTP client)

Until your stripe SDK exposes stable helpers for every v2 path, httpx or requests keeps you aligned with the docs:

import os
import requests

def create_account_v2():
    r = requests.post(
        "https://api.stripe.com/v2/core/accounts",
        headers={
            "Authorization": f"Bearer {os.environ['STRIPE_SECRET_KEY']}",
            "Stripe-Version": "<version-from-stripe-docs>",
            "Content-Type": "application/json",
        },
        json={
            # ...same structure as the cURL example...
        },
        timeout=30,
    )
    r.raise_for_status()
    return r.json()

PHP (Laravel-style)

Use Guzzle or Laravel Http::withHeaders([...])->post(...) with the same URL, Stripe-Version, and JSON body. Keep secrets in config, not in source control.

C# (.NET)

Use HttpClient with StringContent(json, Encoding.UTF8, "application/json") and the same headers. Deserialize the JSON response into your own DTOs that track id, requirements, and configuration.* state.

OAuth / Standard accounts: Stripe currently directs platforms that authenticate with OAuth to connected accounts to continue using v1 for that path. Treat Standard as documented in the v1 guide until your OAuth + v2 story is explicitly supported for your use case—see Accounts v2 and OAuth.

Onboarding links (Express-style flows)

For Express-like experiences you still send users through Stripe-hosted onboarding where the product allows it. With a connected account ID returned from v2 (acct_...), Account Links (v1 resource) remain the usual tool for account_onboarding—the same pattern as the v1 article, but the account value may come from a v2 create. Verify compatibility for your API version in Stripe’s docs.

Python (v1 Account Links API, account id from v2 create):

import stripe
stripe.api_key = os.environ["STRIPE_SECRET_KEY"]

link = stripe.AccountLink.create(
    account=connected_account_id,
    refresh_url="https://yourapp.com/reauth",
    return_url="https://yourapp.com/complete",
    type="account_onboarding",
)

Custom-style integrations on v2

Custom still means: you own KYC collection, ToS acceptance, and ongoing verification. On v2 you express that by:

• Sending complete identity data the API requires.
• Requesting merchant / recipient capabilities your product needs.
• Handling requirements and webhooks the same way you would for Custom on v1—only the payload shape differs.

You may still use tos_acceptance-style fields where the v2 schema maps them; follow Stripe’s v2 reference for exact property names.

Using Accounts as customers (customer_account)

Where you used customer=cus_..., many flows accept customer_account=acct_... for an Account that has the customer configuration. Example pattern from Stripe (conceptual):

curl https://api.stripe.com/v1/setup_intents \
  -u "sk_test_...:" \
  -H "Stripe-Version: <version-from-stripe-docs>" \
  -d customer_account=acct_123 \
  -d "payment_method_types[]=card" \
  -d confirm=true \
  -d usage=off_session

Details and supported objects live under using Accounts as customers.

Checking balances

For many Connect operations, connected account scoping is unchanged. Python:

balance = stripe.Balance.retrieve(stripe_account=account_id)

PHP:

$balance = \Stripe\Balance::retrieve([], ['stripe_account' => $accountId]);

C#:

var balance = await stripe.Balance.GetAsync(
    new BalanceGetOptions(),
    new RequestOptions { StripeAccount = accountId }
);

Confirm in Stripe’s docs whether your v2 account IDs behave identically for every Balance and v1 helper you rely on during migration.

Compliance responsibilities

The Standard / Express / Custom compliance split from the original article still applies who collects KYC and who owns disputes. v2 can reduce duplicate identity collection when you add configurations to an existing Account instead of opening a second Customer record.

Choosing a path

Strategic considerations

• Time-to-market: Standard (when OAuth fits) < Express < Custom.
• API surface: v2 adds configuration discipline—plan for migration from v1, not an eternal split (Stripe discourages maintaining both versions simultaneously). - Reference
• SDKs: Expect to use HTTP for some v2 paths until your language SDK is fully aligned.

Final thoughts

Accounts v2 does not erase Standard / Express / Custom—it repackages how you represent connected users in the API. Start from Connect and the Accounts v2 API, add using Accounts as customers when the same legal entity both sells and buys from your platform, and keep the v1 Connect guide handy for OAuth flows and legacy snippets until you have fully moved.

Either way, you still trade off control, compliance, and complexity—only the object model got a long-overdue upgrade.

“It worked locally. Tests passed. But production still broke.”

If you’re building distributed systems with multiple services and frontends, you’ve likely encountered this (whether using .NET + Angular, Node.js + React, Python + Vue, or any combination):

• A backend change gets deployed
• The frontend suddenly breaks
• No tests warned you

The issue isn’t always logic.

It’s often a broken contract between systems.

The Problem: Silent API Breakages

In a typical setup:

• Service Provider: An API or microservice
• Service Consumer: A frontend, app, or another service
• Communication: JSON over HTTP (or any protocol)

Everything depends on one thing:

The consumer and provider agreeing on what data looks like

A Real Example

Let’s use a .NET API and Angular frontend as an example (though this applies to any tech stack).

API Provider (Initial Version)

public class UserDto
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
}

[HttpGet("{id}")]
public IActionResult GetUser(int id)
{
    return Ok(new UserDto
    {
        Id = 1,
        Name = "Billy",
        Email = "billy@example.com"
    });
}

Consumer Interface (Angular Example)

export interface User {
  id: number;
  name: string;
  email: string;
}

Everything works perfectly.

Then a “Small” Change Happens

public class UserDto
{
    public int Id { get; set; }
    public string FullName { get; set; } // renamed
    public string Email { get; set; }
}

Production Result

user.name //  undefined

The UI breaks.

Why Didn’t Traditional Tests Catch This?

• Unit tests → passed (backend logic is fine)
• Integration tests → passed (they used the old model)
• The API still returns valid JSON

But the contract between systems changed—and traditional tests don’t verify that.

What is Contract Testing?

In Contract Testing, a contract is a formal specification of expected behavior and communication rules between two or more components, services or systems.

Contract testing ensures that your service provider always matches what the consumer expects.

A contract defines:

• Request format
• Response structure
• Required fields
• Data types

In Simple Terms

The consumer (frontend, app, or service) defines expectations The provider (API or service) must satisfy them

Consumer vs Provider

Consumer

• Calls the service/API
• Defines expected structure
• Examples: Angular frontend, React app, mobile app, another microservice

Provider

• Returns the data
• Must not break expectations
• Examples: .NET API, Node.js backend, Python service, GraphQL endpoint

How Contract Testing Works

Instead of relying only on integration tests:

1. Angular defines expectations
2. A contract file is generated
3. .NET verifies the contract

Flow

Consumer Test (e.g., Angular)
      ↓
Generates Contract
      ↓
Saved as JSON/YAML
      ↓
Provider verifies against contract (e.g., .NET API)

Practical Example

(.NET backend + Angular frontend as an example)

Step 1: Define Expectations in Consumer (Angular example)

const expectedUser = {
  id: 1,
  name: "Billy",
  email: "billy@example.com"
};

it("should fetch user correctly", async () => {
  const user = await userService.getUser(1);
  expect(user).toEqual(expectedUser);
});

Generated Contract (Simplified)

{
  "request": {
    "method": "GET",
    "path": "/api/users/1"
  },
  "response": {
    "body": {
      "id": 1,
      "name": "Billy",
      "email": "billy@example.com"
    }
  }
}

Step 2: Verify in Provider (.NET API example)

Install Pact:

dotnet add package PactNet

Provider Test

[Fact]
public void VerifyPact()
{
    var pactVerifier = new PactVerifier();

    pactVerifier
        .ServiceProvider("UserApi", "http://localhost:5000")
        .WithFileSource(new FileInfo("pacts/userapi-angular.json"))
        .Verify();
}

If Provider Breaks the Contract

public string FullName { get; set; }

• The test fails immediately

• You catch the issue before deployment

Where Contract Testing Fits

E2E Tests
(user journeys)

Integration Tests
(service interactions)

Contract Tests 
(API agreements)

Unit Tests
(business logic)

Why This Matters in Real Projects

Safer Refactoring

Change DTOs, schema, or API responses without fear. Contract tests verify nothing broke.

Independent Development

Consumer and provider teams move faster. Changes are caught instantly, not in production.

Faster Debugging

Failures clearly show what broke

Stronger CI/CD Pipelines

Consumer Build → Generate Contract
Provider Build → Verify Contract
Deploy → Only if both pass

This works with any tech stack.

Common Mistakes

Over-Specifying Data

Bad:

"name": "Billy Okeyo"

Good:

"name": "string"

Testing Everything

Only validate fields your frontend actually uses

Ignoring Versioning

Breaking contracts without versioning leads to production issues

Best Practices

Keep Contracts Minimal

Focus only on required fields

Version Your API

/api/v1/users
/api/v2/users

Automate in CI/CD

Contracts should be generated and verified automatically

Use Realistic Data

Avoid unrealistic mocks

Final Takeaway

Unit tests verify logic Integration tests verify systems E2E tests verify user flows

Contract tests verify agreements

“Most production bugs aren’t failures… they’re misunderstandings between systems.”

Contract testing eliminates those misunderstandings.

Examples include:

• Sending emails
• Generating reports
• Processing uploaded files
• Running background analytics
• Sending notifications

Running these tasks during an HTTP request can make your application slow and unreliable.

This is where Celery and Redis come in.

Together they allow you to run background jobs asynchronously without blocking your main application.

The Problem with Synchronous Tasks

Imagine a user submits a request that triggers an operation that takes 10 seconds.

For example:

• Generating a financial report
• Parsing a large document
• Sending multiple emails

If your application processes this synchronously:

User Request → Django → Long Task → Response

The user waits for the entire process to finish.

This leads to:

• Slow responses
• Poor user experience
• Possible request timeouts

Introducing Celery

Celery is a distributed task queue that allows you to run background jobs outside the request-response cycle.

Instead of executing tasks immediately, Django sends the job to a queue.

A worker then processes it asynchronously.

The flow becomes:

User Request
    ↓
Django
    ↓
Queue Task
    ↓
Immediate Response
    ↓
Celery Worker
    ↓
Executes Job

This makes your application fast and scalable.

Why Redis?

Celery requires a message broker to manage task queues.

Redis is commonly used because it is:

• Extremely fast
• Lightweight
• Easy to deploy
• Perfect for queues and caching

Redis stores the tasks until workers pick them up.

Example: Sending Email in the Background

Instead of sending email directly in a Django view:

send_mail(
    "Welcome",
    "Thanks for signing up",
    "noreply@example.com",
    [user.email],
)

You create a Celery task:

from celery import shared_task
from django.core.mail import send_mail

@shared_task
def send_welcome_email(email):
    send_mail(
        "Welcome",
        "Thanks for signing up",
        "noreply@example.com",
        [email],
    )

Then call it asynchronously:

send_welcome_email.delay(user.email)

The user gets an immediate response while the email is processed in the background.

Common Use Cases for Celery

Celery is useful for many production tasks:

Email sending

• Welcome emails
• Notifications
• Password resets

Data processing

• Financial calculations
• AI processing
• Data pipelines

Scheduled tasks

• Daily reports
• Cleaning expired sessions
• Updating analytics

Running Celery in Production

A typical Django production stack might look like this:

Users
  ↓
Nginx
  ↓
Gunicorn
  ↓
Django App
  ↓
Redis (Broker)
  ↓
Celery Workers

Each component plays a role:

• Nginx handles web traffic
• Gunicorn runs Django
• Redis manages task queues
• Celery workers execute background jobs

Scaling Celery Workers

One of Celery’s biggest advantages is scalability.

If tasks increase, you simply add more workers.

celery -A project worker --loglevel=info --concurrency=4

More workers mean faster task processing.

Final Thoughts

Celery and Redis are essential tools for running Django applications at scale.

They allow you to:

• Improve response times
• Handle heavy workloads
• Build scalable architectures
• Run background processing reliably

If your Django application handles tasks that take more than a few seconds, moving them to Celery is one of the best architectural decisions you can make.

Images need previewing. CSV files need parsing. Excel files need validation. Large files need handling. And suddenly your “simple upload” becomes a full feature.

In this guide, we’ll build a practical and production-ready file upload system using:

• Angular (Frontend)
• Django + Django REST Framework (Backend)

We’ll cover:

1. Image uploads with preview
2. CSV uploads and parsing
3. Excel uploads and processing
4. Validation and security best practices

Backend Setup (Django + DRF)

1. Install Dependencies

pip install djangorestframework pillow openpyxl pandas

• pillow → Image processing
• openpyxl → Excel support
• pandas → CSV & Excel parsing

2. Configure Media Files

settings.py

MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

urls.py (project level)

from django.conf import settings
from django.conf.urls.static import static

urlpatterns = [
    # your urls
]

if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

Part 1: Image Upload

Django Model

from django.db import models

class Profile(models.Model):
    name = models.CharField(max_length=100)
    image = models.ImageField(upload_to='profiles/')

Serializer

from rest_framework import serializers
from .models import Profile

class ProfileSerializer(serializers.ModelSerializer):
    class Meta:
        model = Profile
        fields = '__all__'

View

from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser, FormParser
from rest_framework import status

class ProfileUploadView(APIView):
    parser_classes = [MultiPartParser, FormParser]

    def post(self, request):
        serializer = ProfileSerializer(data=request.data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Angular Frontend (Image Upload)

HTML

<input type="file" (change)="onFileSelected($event)" accept="image/*" />
<img *ngIf="previewUrl" [src]="previewUrl" width="200" />
<button (click)="upload()">Upload</button>

Component

selectedFile!: File;
previewUrl: string | ArrayBuffer | null = null;

onFileSelected(event: any) {
  this.selectedFile = event.target.files[0];

  const reader = new FileReader();
  reader.onload = () => {
    this.previewUrl = reader.result;
  };
  reader.readAsDataURL(this.selectedFile);
}

upload() {
  const formData = new FormData();
  formData.append('name', 'Billy');
  formData.append('image', this.selectedFile);

  this.http.post('http://localhost:8000/api/upload/', formData)
    .subscribe(res => console.log(res));
}

Part 2: CSV Upload and Parsing

Django View for CSV

import pandas as pd
from rest_framework.views import APIView
from rest_framework.response import Response
from rest_framework.parsers import MultiPartParser

class CSVUploadView(APIView):
    parser_classes = [MultiPartParser]

    def post(self, request):
        file = request.FILES['file']

        if not file.name.endswith('.csv'):
            return Response({'error': 'Invalid file type'}, status=400)

        df = pd.read_csv(file)

        # Example processing
        total_rows = len(df)
        columns = list(df.columns)

        return Response({
            'rows': total_rows,
            'columns': columns
        })

Angular CSV Upload

uploadCSV(file: File) {
  const formData = new FormData();
  formData.append('file', file);

  this.http.post('http://localhost:8000/api/upload-csv/', formData)
    .subscribe(res => console.log(res));
}

Part 3: Excel Upload (.xlsx)

Django View for Excel

class ExcelUploadView(APIView):
    parser_classes = [MultiPartParser]

    def post(self, request):
        file = request.FILES['file']

        if not file.name.endswith('.xlsx'):
            return Response({'error': 'Invalid file type'}, status=400)

        df = pd.read_excel(file)

        summary = {
            'rows': len(df),
            'columns': list(df.columns)
        }

        return Response(summary)

Validation & Security Best Practices

1. Limit File Size

In settings.py:

DATA_UPLOAD_MAX_MEMORY_SIZE = 5242880  # 5MB

2. Validate File Type Properly

Don’t rely only on extension.

if file.content_type not in ['image/jpeg', 'image/png']:
    return Response({'error': 'Invalid image format'}, status=400)

3. Rename Uploaded Files

import uuid

def upload_to(instance, filename):
    ext = filename.split('.')[-1]
    return f"uploads/{uuid.uuid4()}.{ext}"

4. Handle Large Files with Streaming

For very large CSV files, process in chunks:

for chunk in pd.read_csv(file, chunksize=1000):
    # process chunk
    pass

Bonus: Returning Processed Data

Sometimes you don’t just upload — you process and return a result file.

Example: Generate processed CSV

from django.http import HttpResponse

response = HttpResponse(content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename="processed.csv"'
df.to_csv(response, index=False)
return response

Final Thoughts

File uploads are not just about saving files.

They are about:

• Validation
• Security
• User experience
• Scalability

When done correctly, they become powerful data pipelines inside your application.

If you’re building admin systems, reporting tools, learning platforms, or fintech dashboards — mastering uploads is essential.

One pattern stood out immediately.

I bought a MacBook, and at the time, it felt sufficient. It solved the problem I had: performance, portability, and a solid development environment. From a purely functional standpoint, nothing else was required.

But that sense of “enough” didn’t persist.

Shortly after, I justified getting an iPhone partly for convenience, partly for the ecosystem. Once that was in place, the Apple Watch began to feel like a logical extension: notifications, health tracking, tighter integration. And somehow, even after all that, an Apple TV entered the setup as if the system was still incomplete.

Individually, each purchase was defensible. Collectively, they formed a pattern I hadn’t consciously designed.

What struck me during this reflection was not the spending itself, but the realization that my definition of completeness kept shifting. Satisfaction behaved like a moving target, always one accessory away.

Later, I came across a concept that explained this behavior with surprising precision: The Diderot Effect.

What Is the Diderot Effect?

The Diderot Effect describes a behavioral phenomenon where acquiring a new item triggers a cascade of related purchases, driven by the need for consistency rather than necessity.

The concept originates from Denis Diderot, an 18th-century philosopher who, after receiving a luxurious new robe, felt compelled to replace much of his existing furniture because it no longer “matched” the new standard he had introduced.

In modern terms, the Diderot Effect is essentially scope creep applied to consumption.

Why the Diderot Effect Resonates Strongly in Tech

Technology ecosystems are uniquely optimized to amplify this effect.

1. Ecosystem Lock-In as a Design Strategy

Modern tech products are rarely designed as standalone components. They are built as interdependent systems:

• Phone ↔ laptop ↔ watch ↔ TV
• Cloud services ↔ subscriptions ↔ hardware
• Accessories that unlock “full functionality”

Once you introduce a single high-tier component into your stack, the rest of your setup begins to feel like technical debt.

2. Identity-Driven Tooling

In tech, tools are not just utilities they are signals. Owning certain devices, editors, keyboards, or setups subtly communicates:

• Competence
• Professionalism
• “Seriousness” about the craft

The internal logic becomes:

If I’m this kind of developer, my tools should reflect that.

This is where the Diderot Effect stops being about objects and starts being about identity coherence.

3. Optimization Culture

Engineers are trained to optimize systems. Unfortunately, that mindset often spills into consumption:

• Latency → upgrade
• Minor friction → replace
• Marginal improvement → justify cost

Not every inefficiency needs a hardware solution but the instinct is deeply ingrained.

Real-World Tech Examples of the Diderot Effect

• Buying a new laptop → external monitor → mechanical keyboard → standing desk
• Upgrading a phone → earbuds → watch → chargers in every room
• Improving a dev setup → paid tools → subscriptions → cloud services

Each step feels incremental. The aggregate cost rarely does.

The Hidden Costs Engineers Often Overlook

1. Financial Leakage Through “Reasonable” Decisions

Most Diderot-driven purchases pass basic rational checks. The problem isn’t irrationality it’s unbounded justification.

2. Perpetual Dissatisfaction

When standards continuously shift upward, stability disappears. You’re always upgrading the environment instead of leveraging it.

3. Loss of Intentional System Design

Your setup evolves reactively rather than architecturally.

Managing the Diderot Effect as a Technical Thinker

The solution isn’t abstinence it’s intentional system design.

1. Introduce Decision Latency

Treat secondary purchases like production changes:

• Add a delay (48 hours, a week, or a sprint)
• Re-evaluate the actual requirement

Most impulses decay with time.

2. Define System Boundaries

Before upgrading, explicitly define the scope:

“This purchase solves this problem. Nothing else changes.”

Boundaries prevent cascade failures.

3. Separate Functional Debt from Aesthetic Debt

Ask:

• Is this blocking productivity?
• Or does it simply look inferior relative to a new baseline?

Most Diderot chains are triggered by aesthetics masquerading as inefficiency.

4. Budget Upgrades as Projects, Not Reactions

Instead of ad-hoc improvements:

• Plan upgrades quarterly or annually
• Allocate fixed budgets
• Close the project when scope is met

No silent expansions.

5. Be Wary of Identity-Based Justifications

Phrases like:

• “At my level, I should have…”
• “This setup doesn’t reflect where I am now…”

These are signals that the Diderot Effect is driving the decision, not necessity.

When the Diderot Effect Can Be Used Intentionally

Not all cascades are bad.

Applied deliberately, the effect can reinforce positive systems:

• Fitness → nutrition → sleep optimization
• Learning → better tools → deeper focus
• Workspace upgrades that genuinely reduce friction

The difference lies in conscious orchestration versus unconscious drift.

Final Thoughts

The Diderot Effect is not a failure of discipline, it’s a predictable system behavior triggered by introducing a higher standard into an existing environment.

In tech, where optimization and cohesion are prized, this effect is especially potent.

Reflecting on my own experience, from a single MacBook purchase to a fully integrated ecosystem made one thing clear: “Enough” must be explicitly defined, or it will keep moving.

As engineers, we design systems for a living. Our consumption habits deserve the same level of intentionality.

If you told me at the beginning of 2025 that I’d spend the year writing about scalable systems, testing strategies, outages, performance optimizations, and developer excuses I would’ve believed you. If you also told me I’d still be debugging things that “worked yesterday,” I would’ve believed you even faster.

This article is a recap of everything I wrote this year: the lessons, the wins, the bugs, and the “how did this even pass code review?” moments. Think of it as a Spotify Wrapped, but for technical articles, minus the judgment (okay, maybe a little).

System Design & Scalability — Because Your App Will Grow (Whether You’re Ready or Not)

Summary: This year, I spent a lot of time talking about scalable backend systems — not because every app needs to handle millions of users, but because every app eventually meets its first “why is the server on fire?” moment.

• How to Build Scalable Backend Systems with Python, C#, PHP and Dart A practical guide to designing systems that won’t collapse the moment your app gets featured on Twitter… or worse, WhatsApp groups.

Key lesson: Scalability isn’t about overengineering — it’s about not regretting your life choices later.

Testing — Trust Issues, But Make Them Automated

Summary: Testing dominated a good part of the year, mainly because nothing builds trust issues faster than code that looks correct but isn’t.

This series walked through testing from the basics to full CI/CD integration — because “it works on my machine” is not a deployment strategy.

• Unit, Integration, and End-to-End Tests — Part 1 Where we learn that tests are not the enemy — flaky tests are.
• Unit Testing in Depth — Part 2 Small tests, big confidence.
• Integration Testing in Depth — Part 3 When components finally talk to each other… and start arguing.
• End-to-End Testing in Depth — Part 4 Testing your app the same way users break it.
• The Testing Pyramid & CI/CD — Part 5 The moment you realize automation saves both time and sanity.

Key lesson: Tests don’t slow you down — debugging production issues does.

Performance & Tooling — Making Code Faster Without Sacrificing Sleep

Summary: Performance optimization showed up in different forms this year — from modern runtimes to build optimizations. Because sometimes the app isn’t slow… it’s just doing unnecessary work very enthusiastically.

• Diving into WebAssembly: What It Is and Why It Matters For when JavaScript alone just isn’t fast enough.
• Tree Shaking in TypeScript Removing code you forgot existed but was still shipped to production.

Key lesson: If users say your app is slow, they’re probably right. The logs just haven’t confessed yet.

Engineering Culture — Because Code Is Written by Humans (Flawed Ones)

Summary: Not everything this year was serious architecture talk. Some articles leaned into the very human side of software development — where excuses are plentiful and accountability is… negotiable.

• Top 10 Developer Excuses When Code Breaks A humorous breakdown of things we’ve all said — and what actually went wrong.

Key lesson: If you’ve never blamed the cache, the network, or “some weird edge case,” are you even a developer?

Industry Case Studies — Learning the Hard Way (So You Don’t Have To)

Summary: One of the most impactful pieces this year was breaking down a real-world outage — because nothing teaches engineering humility like watching large systems fail in creative ways.

• What Engineers Can Learn From the Cloudflare Outage A reminder that one bad configuration can humble the biggest companies.

Key lesson: Distributed systems don’t fail loudly — they fail politely, globally, and at the worst possible time.

What 2025 Really Taught Me

If I had to summarize the year in engineering truths:

• Scalability matters before users complain.
• Tests are cheaper than apologies.
• Performance issues hide in plain sight.
• Systems fail — preparation determines whether you panic or recover.
• Developers are predictable creatures with unpredictable bugs.

Looking Ahead

In the next year, expect:

• More real-world case studies
• Deeper dives into distributed systems and cloud patterns
• Practical articles you can actually apply on Monday morning

Thank you for reading, sharing, and occasionally finding bugs in my examples (yes, I see you).

Here’s to another year of writing code, fixing mistakes, and pretending we knew what we were doing all along.

On November 18, 2025, the internet shook a little.

Cloudflare, the massive networking and security platform powering millions of websites globally, experienced a major outage that resulted in widespread 5xx errors across the internet. For several hours, key services like CDN routing, Workers KV, Access authentication, and even Cloudflare’s own dashboard were degraded.

In their official incident report, Cloudflare broke down exactly what happened, and the root cause was surprisingly small and deceptively simple:

A configuration file grew larger than expected, violated a hidden assumption in the proxy code, and triggered runtime panics across the global edge.

This incident is a powerful case study in modern distributed systems. Let’s break down what went wrong — and why engineers everywhere should take note.

What Actually Happened? A Chain Reaction From One Oversized File

The root cause began with a permissions change in Cloudflare’s ClickHouse database cluster, which led to duplicate rows in a dataset used by their Bot Management engine. That duplication caused the generated “feature file”, a config-like file that proxies rely on to double in size.

Here’s where assumptions came back to haunt them:

• Cloudflare’s proxy engine expected a maximum of 200 features.
• The new file exceeded that limit.
• An .unwrap() in their Rust-based FL2 proxy assumed the file would always be valid.
• That assumption failed — the code panicked — resulting in cascading 5xx failures.

As Cloudflare noted in their report, this caused two types of breakage across their edge:

1. FL2 proxies (new engine): Panicked → produced 5xx errors
2. FL proxies (old engine): Failed to process bot scores → defaulted to zero → broke logic in Access, rules, authentication, and security products

To make things even more confusing, Cloudflare’s status page (hosted externally) briefly went offline too, creating a misleading early hypothesis that the outage was a massive DDoS attack.

It wasn’t. It was configuration drift.

Why This Seemingly Small Bug Became a Big Internet Event

Config files have become “just another part of the deployment pipeline,” especially in cloud platforms where machine-generated metadata drives features. But Cloudflare’s outage shows:

• Config is not static
• Config can be corrupted
• Config needs validation just like code

Because this file was distributed globally across tens of thousands of Cloudflare servers, a single flawed generation step caused a worldwide issue within minutes.

Distributed systems amplify mistakes — both good ones and bad ones.

Key Engineering Lessons We Should All Learn

1. Never rely on hidden assumptions

Cloudflare’s proxy code assumed the feature file would never exceed a certain size. That “should never happen” moment is often the birthplace of outages.

Lesson: Add explicit limits, schema checks, and sanity validations to all config ingestion paths.

2. Configuration is part of your software supply chain

The feature file was generated, replicated, and consumed automatically — no human intervention. That makes it just as risky as code.

Lesson: Treat configuration pipelines as first-class citizens: test them, validate them, gate them.

3. Build for graceful degradation, not hard crashes

A single .unwrap() took down parts of the internet.

Lesson: Fail softly. If config is invalid, degrade safely, skip rules, or revert to defaults — don’t panic.

4. Feature flags and kill switches are essential

Cloudflare themselves acknowledged the need for a more robust kill-switch system in their follow-up plans.

Lesson: Every modern engineering team should have:

• global feature kill switches
• fast configuration rollback
• manual override options

5. Monitoring needs context, not just alarms

Many engineers watching the outage thought it was an external attack. Alerting didn’t tell them where the failure was coming from, just that everything was “down.”

Lesson: Monitoring should distinguish between:

• origin failure
• edge failure
• config failure
• auth failure
• internal propagation issues

Context reduces misdiagnosis.

6. Observability tools must be lightweight during crises

During recovery, Cloudflare reported that their debugging systems started consuming high CPU, which slowed other mission-critical services.

Lesson: Your troubleshooting tools must use minimal resources when the system is stressed.

7. Transparency helps the whole industry learn

Cloudflare’s detailed post-mortem is a model for engineering culture. Their openness helps engineers worldwide understand real failure modes in large-scale systems.

Lesson: Share your failures. They help others avoid the same mistakes.

The Bigger Picture: Why This Incident Matters

Cloudflare’s outage wasn’t just a story about a config file. It was a reminder of how fragile the modern internet can be:

• We automate everything
• We trust our pipelines
• We deploy continuously
• We assume schemas won’t change
• We rely on millions of machines making the same decision correctly

But when the assumptions underneath those systems break, failures propagate at machine speed.

For engineers, SREs, DevOps teams, and platform architects, this incident underscores a fundamental truth:

Your system is only as resilient as your least-validated assumption.

Final Thoughts

Cloudflare’s outage was a blip in the timeline of the internet, but the lessons are timeless.

Distributed systems will always fail, the question is how gracefully they fail, how quickly they recover, and how deeply the organization learns from the event.

Cloudflare’s transparency, analysis, and remediation steps set a strong example for engineering teams everywhere.

Update (2026): Stripe’s Accounts v2 API uses a unified Account with configurations (merchant, customer, recipient) so one identity can sell, pay your platform, and receive transfers without parallel Customer IDs. For a full guide in the same spirit as this article—but aligned to v2—see Stripe Connect on Accounts v2. Everything below remains the reference for Accounts v1 (Account.create, OAuth, Account Links as shown). V2 though is the recommended path for the new Stripe Connect Integrations.

Stripe offers a powerful ecosystem for building payment platforms, marketplaces, and financial applications. One of its most powerful products, Stripe Connect, allows you to facilitate payments between multiple parties while maintaining flexibility over branding, compliance, and user experience.

Choosing between Standard, Express, and Custom accounts is one of the most important architectural decisions you’ll make when designing a payment platform. This guide combines a step-by-step tutorial and deep technical analysis to help you understand and implement each type.

What Is Stripe Connect?

Stripe Connect is designed for platforms that process payments on behalf of others. It enables you to connect user accounts (called Connected Accounts) to your Platform Account so you can:

• Collect payments from customers
• Distribute payouts to vendors, lenders, or service providers
• Optionally collect platform fees

Your platform never needs to hold funds directly (unless you design it to), and Stripe handles the movement of money under the hood.

Connected Accounts Overview

Each connected account represents a business, seller, or user on your platform. Depending on how much control and responsibility you want, you’ll choose between:

1. Standard — Stripe handles almost everything.
2. Express — Stripe handles compliance, you handle limited UX.
3. Custom — You handle everything; Stripe is invisible to your users.

Architectural Overview

All three integration types share a common flow:

Customer → Your Platform → Connected Account → Bank Payout

However, the control points differ:

• With Standard, Stripe owns the UX and dashboard.
• With Express, Stripe hosts onboarding and dashboard, but your platform manages relationships.
• With Custom, your platform builds everything: UI, onboarding, payouts, and reporting.

Step-by-Step Implementation by Account Type

Let’s look at how to implement each account type with C#, Laravel (PHP) and Python examples and understand their implications.

Standard Accounts — Easiest Setup, Minimal Control

Concept

Standard accounts are the simplest to integrate. Users connect their own existing Stripe accounts using OAuth. Stripe handles compliance, payouts, and reporting. Your platform receives a small application fee from each transaction.

Use Case

Ideal for marketplaces or SaaS platforms where users already have Stripe accounts and prefer to manage their own dashboards.

Implementation

C#

var accountLink = await stripe.AccountLinks.CreateAsync(new AccountLinkCreateOptions {
    Account = connectedAccountId,
    RefreshUrl = "https://your-platform.com/reauth",
    ReturnUrl = "https://your-platform.com/success",
    Type = "account_onboarding",
});

Laravel (PHP)

$accountLink = \Stripe\AccountLink::create([
    'account' => $connectedAccountId,
    'refresh_url' => route('stripe.reauth'),
    'return_url' => route('stripe.success'),
    'type' => 'account_onboarding',
]);

Python

import stripe
stripe.api_key = "sk_test_..."

account_link = stripe.AccountLink.create(
    account=connected_account_id,
    refresh_url="https://your-platform.com/reauth",
    return_url="https://your-platform.com/success",
    type="account_onboarding"
)

Pros and Cons

Pros

• Minimal setup
• Stripe handles everything (KYC, compliance, payouts)
• Reduced liability

Cons

• Limited branding control
• Users must leave your platform to manage payments
• Harder to create a unified experience

Express Accounts — Fast Onboarding, Shared Control

Concept

Express accounts strike a balance between simplicity and control. You manage account creation and linking, but Stripe handles onboarding and provides a lightweight dashboard for your users.

Use Case

Perfect for gig or service platforms (like driver or freelancer apps) where quick onboarding and basic payout visibility are key.

Implementation

C#

var account = await stripe.Accounts.CreateAsync(new AccountCreateOptions {
    Type = "express",
    Country = "US",
    Email = "user@example.com"
});

var link = await stripe.AccountLinks.CreateAsync(new AccountLinkCreateOptions {
    Account = account.Id,
    RefreshUrl = "https://yourapp.com/reauth",
    ReturnUrl = "https://yourapp.com/complete",
    Type = "account_onboarding"
});

Laravel (PHP)

$account = \Stripe\Account::create([
    'type' => 'express',
    'country' => 'US',
    'email' => $request->input('email'),
]);

$link = \Stripe\AccountLink::create([
    'account' => $account->id,
    'refresh_url' => route('stripe.reauth'),
    'return_url' => route('stripe.success'),
    'type' => 'account_onboarding',
]);

Python

account = stripe.Account.create(
    type="express",
    country="US",
    email="user@example.com"
)

link = stripe.AccountLink.create(
    account=account.id,
    refresh_url="https://yourapp.com/reauth",
    return_url="https://yourapp.com/complete",
    type="account_onboarding"
)

Pros and Cons

Pros

• Faster onboarding
• Stripe handles compliance and verification
• Users get access to payout history via the Express dashboard

Cons

• Limited customization
• Stripe branding remains visible
• Less flexibility over reporting or custom payout logic

Custom Accounts — Full Control, Full Responsibility

Concept

Custom accounts are designed for white-labeled platforms. Your app controls everything: onboarding, KYC collection, balances, and payouts. Stripe is completely invisible to the end user.

Use Case

Ideal for fintech, embedded finance, lending, or any system that requires deep integration and custom user experiences.

Implementation

C#

var accountOptions = new AccountCreateOptions {
    Type = "custom",
    Country = "US",
    Email = data.Email.ToLower(),
    Capabilities = new AccountCapabilitiesOptions {
        CardPayments = new AccountCapabilitiesCardPaymentsOptions { Requested = true },
        Transfers = new AccountCapabilitiesTransfersOptions { Requested = true }
    },
    TosAcceptance = new AccountTosAcceptanceOptions {
        Date = DateTimeOffset.UtcNow.ToUnixTimeSeconds(),
        Ip = httpContext.Connection.RemoteIpAddress.ToString()
    }
};

var account = await stripe.Accounts.CreateAsync(accountOptions);

Laravel (PHP)

$account = \Stripe\Account::create([
    'type' => 'custom',
    'country' => 'US',
    'email' => strtolower($data['email']),
    'capabilities' => [
        'card_payments' => ['requested' => true],
        'transfers' => ['requested' => true],
    ],
    'tos_acceptance' => [
        'date' => time(),
        'ip' => request()->ip(),
    ],
]);

Python

account = stripe.Account.create(
    type="custom",
    country="US",
    email=data["email"].lower(),
    capabilities={
        "card_payments": {"requested": True},
        "transfers": {"requested": True},
    },
    tos_acceptance={
        "date": int(time.time()),
        "ip": request.remote_addr,
    }
)

Fund Flow Example

+--------------------------------------+
|             Platform (You)           |
|    No direct money handling          |
+--------------------+-----------------+
                     |
                     ▼
          Create & Manage Connected Accounts
                     |
   +--------------------------------------+
   |     Connected Account (Business)     |
   |  💵 Has Stripe balance, bank info    |
   |  🧾 Disburses and collects funds     |
   +--------------------------------------+

Money Movement

• IN: Customer → Connected Account (via ACH/Card)
• OUT: Connected Account → Bank (payouts)
• Platform orchestrates, Stripe moves the money

Checking Balances

Python

balance = stripe.Balance.retrieve(stripe_account=account_id)

Laravel (PHP)

$balance = \Stripe\Balance::retrieve([], ['stripe_account' => $accountId]);

C#

var balance = await stripe.Balance.GetAsync(
    new BalanceGetOptions(),
    new RequestOptions { StripeAccount = accountId }
);

Compliance Responsibilities

Tip: For Custom accounts, implement Stripe Identity, webhooks, and Stripe Radar to automate verification and fraud detection.

Choosing the Right Integration Type

Strategic Considerations

• Time-to-market: Standard < Express < Custom
• Compliance load: Stripe-heavy → Custom-heavy
• Branding control: Minimal → Full
• Revenue potential: Low (Standard) → High (Custom)

Final Thoughts

Every Stripe Connect integration represents a trade-off between control, compliance, and complexity:

• Standard — easiest to deploy, lowest control
• Express — balanced control and simplicity
• Custom — ultimate flexibility with greater responsibility

If your goal is speed, start with Express. If your goal is brand control and scalability, build with Custom.

Either way, Stripe Connect gives you a future-proof foundation for managing payments, onboarding users, and creating rich financial experiences all through powerful APIs available across languages.

In this final part of our testing series, we’ll:

• Understand the Testing Pyramid model.
• Explore how to balance different types of tests.
• Learn how to integrate testing into CI/CD pipelines.
• Review best practices for building a sustainable testing culture.

The Testing Pyramid Explained

The Testing Pyramid (popularized by Mike Cohn) is a metaphor for structuring automated tests:

        ▲
        |   End-to-End (fewest, slowest)
        |
        |   Integration (moderate amount, balanced)
        |
        |   Unit Tests (largest base, fastest)
        ▼

1. Unit Tests (Foundation)

• Fast, cheap, and precise.
• Cover small pieces of logic in isolation.
• Should make up 60–70% of your automated test suite.

Example: Testing a function that calculates interest rates.

2. Integration Tests (Middle Layer)

• Validate how components interact.
• Cover database queries, API requests, or service orchestration.
• Should make up 20–30% of your test suite.

Example: Ensuring your API endpoint correctly fetches data from the database and formats the response.

3. End-to-End Tests (Top)

• Simulate real user flows across the full stack.
• Catch bugs unit or integration tests can’t.
• Should make up 10–15% of your suite (because they’re slow and expensive).

Example: A test where a user logs in, adds a product to their cart, and completes checkout.

Integrating Tests into CI/CD Pipelines

Automated tests are only valuable if they’re run consistently. Modern software teams embed them into CI/CD pipelines.

Step 1: Run Unit Tests Early

• Execute on every commit or pull request.
• Fail fast: block merges if unit tests fail.

Step 2: Run Integration Tests on Build/Deploy

• Run after the app compiles successfully.
• Can use a containerized test environment with mock or staging databases.

Step 3: Run End-to-End Tests on Staging

• Triggered before production deployment.
• Some teams run smoke E2E tests in production (carefully) to ensure critical flows still work.

Example CI/CD Flow (GitHub Actions / GitLab / Jenkins):

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install dependencies
        run: npm install
      - name: Run unit tests
        run: npm test -- --unit
      - name: Run integration tests
        run: npm test -- --integration
      - name: Run e2e tests
        run: npm run test:e2e

Best Practices for a Balanced Testing Strategy

1. Follow the Pyramid, Not the Ice Cream Cone

• Too many E2E tests = slow, brittle pipeline.
• Too few unit tests = shaky foundation.
• Balance is key.

2. Use Test Doubles Wisely

• Mocks/Stubs in unit tests to isolate dependencies.
• Minimal mocking in integration tests — rely on real services where possible.

3. Make Tests Deterministic

• Tests should always produce the same result.
• Avoid flaky tests (caused by race conditions, timeouts, or external dependencies).

4. Keep Tests Fast

• Aim for seconds, not minutes.
• Developers won’t run slow tests locally.

5. Automate Everything in CI/CD

• Manual testing has its place, but regression checks must be automated.
• CI/CD pipelines should be your safety net before production.

6. Monitor and Improve Test Coverage

• Coverage isn’t everything, but low coverage usually indicates gaps.
• Focus on critical paths rather than chasing 100%.

7. Treat Tests as Code

• Tests should be maintainable, reviewed, and refactored like production code.
• Avoid “test rot” where tests become outdated or ignored.

Final Thoughts

The journey from unit → integration → E2E tests gives you confidence at every level of your system.

• Unit tests keep your building blocks solid.
• Integration tests ensure the blocks fit together.
• E2E tests confirm the entire structure works as intended.

By following the Testing Pyramid, integrating tests into CI/CD pipelines, and practicing discipline in writing effective tests, you’ll achieve:

• Faster feedback loops.
• Fewer production bugs.
• Higher developer confidence.
• Happier end-users.

Testing isn’t just about catching bugs, it’s about building trustworthy software at scale.

Next Steps for You:

• Audit your current test suite.
• See if you’re over-relying on one type of test.
• Gradually reshape your suite into a pyramid, not an ice cream cone.

E2E testing is about validating the system as a whole, covering everything from frontend to backend, database, APIs, and sometimes even external services.

What Are E2E Tests?

E2E tests replicate user behavior and verify that the system responds correctly. Think of them as automated users interacting with your app.

Example workflows tested in E2E:

• User logs in → Dashboard loads correctly.
• User creates a record → It’s stored in DB → Appears in the UI.
• Checkout flow in an e-commerce app → Product added → Payment processed → Order confirmed.

Why E2E Tests Matter

• Validate Real Workflows → Ensure the system behaves like users expect.
• Catch System-Wide Failures → Config issues, routing errors, DB schema mismatches.
• Test Critical Paths → Login, payments, onboarding, search, etc.
• Provide Business Confidence → Stakeholders see features working as intended.

Characteristics of E2E Tests

• High Coverage – Test across the full stack.
• Slowest of All Tests – Often run in CI/CD pipelines, not during active coding.
• Brittle – Small UI changes can break tests, so balance is key.
• Mimic Real User Behavior – Often browser-driven or API-driven.

Side-by-Side Examples

Scenario:

A user visits a web app, logs in, and sees their profile page with their name.

Python (Selenium + pytest)

# test_e2e_login.py
from selenium import webdriver
from selenium.webdriver.common.by import By
import pytest

@pytest.fixture
def driver():
    driver = webdriver.Chrome()
    yield driver
    driver.quit()

def test_login_flow(driver):
    driver.get("http://localhost:5000/login")

    driver.find_element(By.NAME, "username").send_keys("alice")
    driver.find_element(By.NAME, "password").send_keys("password123")
    driver.find_element(By.ID, "login-button").click()

    profile_name = driver.find_element(By.ID, "profile-name").text
    assert profile_name == "Alice"

C# (Selenium + xUnit)

using Xunit;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;

public class LoginE2ETests
{
    [Fact]
    public void Login_ShowsProfilePage()
    {
        using var driver = new ChromeDriver();
        driver.Navigate().GoToUrl("http://localhost:5000/login");

        driver.FindElement(By.Name("username")).SendKeys("alice");
        driver.FindElement(By.Name("password")).SendKeys("password123");
        driver.FindElement(By.Id("login-button")).Click();

        var profileName = driver.FindElement(By.Id("profile-name")).Text;
        Assert.Equal("Alice", profileName);
    }
}

TypeScript (Playwright / Cypress)

(Playwright example below — Cypress would be very similar)

// login.e2e.test.ts
import { test, expect } from "@playwright/test";

test("user can login and see profile page", async ({ page }) => {
  await page.goto("http://localhost:3000/login");

  await page.fill("input[name=username]", "alice");
  await page.fill("input[name=password]", "password123");
  await page.click("#login-button");

  const profileName = await page.textContent("#profile-name");
  expect(profileName).toBe("Alice");
});

PHP (Laravel Dusk for browser automation)

// tests/Browser/LoginTest.php
<?php

namespace Tests\Browser;

use Laravel\Dusk\Browser;
use Tests\DuskTestCase;

class LoginTest extends DuskTestCase
{
    public function testUserCanLoginAndSeeProfile()
    {
        $this->browse(function (Browser $browser) {
            $browser->visit('/login')
                    ->type('username', 'alice')
                    ->type('password', 'password123')
                    ->press('Login')
                    ->assertSee('Alice');
        });
    }
}

Comparison Table

Best Practices for Writing Effective E2E Tests

1. Test Critical User Journeys Only
   • Focus on login, checkout, payments, onboarding, etc.
   • Don’t waste time E2E testing every small feature.
2. Keep Tests Deterministic
   • Avoid flakiness by controlling test data.
   • Seed databases with known test records.
3. Use Test-Specific Environments
   • Run E2E tests in staging or CI environments.
   • Isolate from production data and services.
4. Clean Up After Tests
   • Ensure created records (users, orders) are rolled back or deleted.
5. Use IDs and Stable Selectors
   • Avoid brittle selectors like div:nth-child(3).
   • Use unique IDs or data-test attributes.
6. Parallelize and Optimize
   • Run tests in parallel (e.g., Playwright, Cypress).
   • Keep them minimal to avoid bloated CI runs.
7. Combine with Unit & Integration Tests
   • Don’t rely solely on E2E.

   • Use a test pyramid strategy:

     • 70% Unit Tests
     • 20% Integration Tests
     • 10% E2E Tests
8. Monitor and Review Regularly
   • E2E tests can get brittle.
   • Review and refactor when the UI or workflows change.

Key Takeaways

• E2E tests replicate the user’s journey and validate the full system.
• They are slower and more brittle, but they give the highest level of confidence.
• Use them sparingly for critical paths (login, payments, core workflows).

• Balance your test pyramid:

  • Many unit tests
  • Fewer integration tests
  • Very few but powerful E2E tests
• Choose the right tools for your stack (Selenium, Playwright, Cypress, Dusk).
• Follow best practices to keep tests reliable and maintainable.

This post is a practical, language-agnostic guide to integration testing, plus side-by-side, runnable patterns for Python, C# (.NET), TypeScript (Node/Express) and PHP (Laravel) so you can immediately apply the ideas in your stack.

What integration tests are (and are not)

Integration tests verify behavior across component boundaries, multiple classes, modules, services or infrastructure pieces that would not be exercised by a unit test.

They are not:

• A replacement for unit tests (they’re slower & coarser).
• Full UI-driven E2E tests (unless you intentionally include the UI).

They are good for:

• Verifying DB reads/writes via your data access layer.
• Testing service-to-service interactions.
• Ensuring message queue jobs and workers together produce expected state.
• Checking how your app handles external API payloads (with a mock or stub of that API).

Goals & tradeoffs

Goals

• Catch bugs that only appear when components are wired together.
• Validate API contracts inside your own system.
• Give more realistic coverage than unit tests.

Tradeoffs

• Slower than unit tests.
• Harder to make fully deterministic (external services, timing).
• Need careful setup/teardown to stay reliable.

Core patterns & recommendations

1. Use realistic but controlled dependencies

• Prefer a real database (or the same engine, e.g., PostgreSQL) rather than mocking DB calls.
• For external services (payment gateways, email providers), use service doubles: a local mock server, WireMock, or HTTP interceptors (nock, responses, Http::fake). Don’t call the live service in CI.

2. Isolate tests

• Run each test in a transaction and roll it back (if possible), or recreate schema between tests.
• Or give each test its own ephemeral database (unique DB name/per-worker) when running tests in parallel.

3. Keep tests focused

• Each integration test should exercise a meaningful interaction or flow (e.g., API -> DB, or API -> external-service-stub -> DB), not every possible path.

4. Seed deterministic test data

• Use builders/fixtures to create known state. Avoid random data unless seeded.

5. Manage long-running processes

• For queues/workers, either run workers synchronously in tests, use a fake queue, or spin up a test worker process in CI.

6. Use testcontainers or docker-compose in CI

• For close-to-production fidelity, use Testcontainers (or docker-compose) to provision real DBs and services in CI.

7. Avoid flaky tests

• No sleeps/time-based races. Use blocking signals, polling with timeouts, or deterministic stubs.

When to mock vs when to use real services

• Real DB: Prefer real DB engine (Postgres, MySQL). SQLite is OK for many cases but can mask engine-specific issues.
• External APIs: Mock in integration tests. Use contract testing (Pact) to keep mocked expectations in-sync.
• Caches/Queues: Use in-memory or test doubles unless you must validate the actual middleware behavior.

Observability: make debugging failing integration tests easy

• Emit structured logs during tests (include request IDs).
• Capture and print responses and DB state on failure.
• Keep helpful assertion messages.

Checklist: test lifecycle

1. Create test environment (DB, migrations applied).
2. Seed minimal deterministic data.
3. Execute action via real interfaces (HTTP client, direct call).
4. Assert state persisted, side effects happened (e.g., DB row created, message pushed, HTTP call stubbed).
5. Tear down (transaction rollback, truncate tables, drop DB).

Common pitfalls & fixes

• Flaky tests: avoid sleep-based waiting; use retry-with-timeout polling and assert deterministically.
• Slow setup: keep per-test setup minimal; use transactional rollback where possible.
• Parallel test collisions: give tests separate DBs or use unique table prefixes.
• “Works locally but fails in CI”: mirror CI environment locally using Docker/Testcontainers and run tests there.

Example integration tests (side-by-side)

Scenario used across examples:

A POST /users endpoint that creates a user record in the database and triggers an HTTP call to an external email service (welcome email). Integration test will create a user via HTTP, verify DB row exists, and verify the email call was made (mocked).

Python — FastAPI + SQLAlchemy + pytest + requests-mock

Notes: Use sqlite:///:memory: or a Testcontainers Postgres for higher fidelity in CI. requests-mock stubs outgoing HTTP calls.

# app.py (pseudo)
from fastapi import FastAPI, Depends
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

app = FastAPI()

# App factory to pass different DB URLs in tests
def create_app(db_url, email_service_url):
    engine = create_engine(db_url)
    Session = sessionmaker(bind=engine)
    # create tables...
    app.state.db = Session
    app.state.email_url = email_service_url

    @app.post("/users")
    def create_user(payload: dict):
        sess = app.state.db()
        user = User(name=payload["name"], email=payload["email"])
        sess.add(user); sess.commit()
        # Send welcome email via requests.post(app.state.email_url, json=...)
        return {"id": user.id}

    return app

# test_integration.py
import pytest
from fastapi.testclient import TestClient
import requests_mock
from app import create_app

@pytest.fixture
def client(tmp_path):
    # Use sqlite in-memory for speed or file DB for persistence across app
    app = create_app("sqlite:///:memory:", "http://email.test/send")
    client = TestClient(app)
    yield client

def test_create_user_and_send_email(client):
    with requests_mock.Mocker() as m:
        m.post("http://email.test/send", status_code=200, json={"ok": True})
        resp = client.post("/users", json={"name":"Alice","email":"a@example.com"})
        assert resp.status_code == 200
        user_id = resp.json()["id"]

        # Verify DB row exists (open a session)
        Session = client.app.state.db
        sess = Session()
        user = sess.query(User).filter_by(id=user_id).one_or_none()
        assert user is not None
        assert user.email == "a@example.com"

        # Verify external email call occurred
        assert m.called
        assert m.request_history[0].json() == {"to": "a@example.com", "template": "welcome"}

Tips

• For CI, replace sqlite with Testcontainers Postgres: create_app(postgres_url, ...).
• Use DB migrations in setup if using a real DB.

C# (.NET) — ASP.NET Core + WebApplicationFactory + InMemory DB / Testcontainer

Notes: Use WebApplicationFactory<TEntryPoint> to spin the app in tests and override service registrations for test doubles.

// In Startup.cs, app reads EmailService via IEmailService (HttpEmailService in prod)

public class TestEmailService : IEmailService {
    public List<EmailMessage> Sent = new();
    public Task SendAsync(EmailMessage msg) { Sent.Add(msg); return Task.CompletedTask; }
}

// Integration test
public class UsersIntegrationTests : IClassFixture<WebApplicationFactory<Program>> {
    private readonly WebApplicationFactory<Program> _factory;

    public UsersIntegrationTests(WebApplicationFactory<Program> factory) {
        _factory = factory.WithWebHostBuilder(builder => {
            builder.ConfigureServices(services => {
                // Replace real DB with in-memory or Testcontainer; replace email service with TestEmailService
                services.AddSingleton<IEmailService, TestEmailService>();
                // Configure EF Core to use InMemoryDatabase or connection string from Testcontainers
            });
        });
    }

    [Fact]
    public async Task PostUsers_CreatesUser_And_SendsEmail() {
        var client = _factory.CreateClient();
        var content = new StringContent("{\"name\":\"Bob\",\"email\":\"b@ex.com\"}", Encoding.UTF8, "application/json");
        var resp = await client.PostAsync("/users", content);
        resp.EnsureSuccessStatusCode();

        // Verify DB: use scope to resolve DbContext
        using(var scope = _factory.Services.CreateScope()) {
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
            var user = db.Users.Single(u => u.Email == "b@ex.com");
            Assert.NotNull(user);
        }

        // Verify TestEmailService captured message
        var emailService = _factory.Services.GetRequiredService<IEmailService>() as TestEmailService;
        Assert.Single(emailService.Sent);
        Assert.Equal("welcome", emailService.Sent[0].Template);
    }
}

Tips

• For a real DB in CI, use Testcontainers .NET to spin up Postgres and set EF Core connection string.
• Overriding services avoids brittle HTTP stubbing, and keeps assertions in-process.

TypeScript (Node/Express) — supertest + sqlite in-memory + nock

Notes: supertest issues HTTP requests to your Express app instance. Use nock to intercept outgoing HTTP.

// app.ts (pseudo)
import express from "express";
import bodyParser from "body-parser";
import { initDb } from "./db";

export function createApp(dbPath: string) {
  const app = express();
  app.use(bodyParser.json());
  const db = initDb(dbPath); // e.g., sqlite3 in-memory or file
  app.post("/users", async (req, res) => {
    const { name, email } = req.body;
    const id = await db.run("INSERT INTO users(name,email) VALUES(?,?)", [name, email]);
    // call external email service via fetch/http client
    await fetch("http://email.test/send", { method: "POST", body: JSON.stringify({ to: email, template: "welcome" }) });
    res.json({ id });
  });
  return app;
}

// test/integration.test.ts
import request from "supertest";
import nock from "nock";
import { createApp } from "../app";
import { openDb, getUserById } from "../db";

describe("POST /users", () => {
  it("creates a user and calls email service", async () => {
    const app = createApp(":memory:");
    const email = nock("http://email.test")
      .post("/send", (body) => body.to === "c@ex.com" && body.template === "welcome")
      .reply(200, { ok: true });

    const res = await request(app)
      .post("/users")
      .send({ name: "Carol", email: "c@ex.com" })
      .expect(200);

    // verify DB
    const user = await getUserById(res.body.id);
    expect(user.email).toBe("c@ex.com");

    // verify external call was made
    expect(email.isDone()).toBe(true);
  });
});

Tips

• For complex schemas, use migrations in test setup or run a dedicated test DB with sqlite file per test.
• For Postgres in CI, spin up DB via docker-compose or Testcontainers Node.

PHP (Laravel) — HTTP tests + RefreshDatabase + Http::fake()

Notes: Laravel has excellent integration testing helpers. RefreshDatabase runs migrations and transacts where possible. Use Http::fake() to intercept external HTTP.

// routes/api.php
Route::post('/users', [UserController::class, 'store']);

// UserController->store uses User model and Http::post('http://email.test/send', ...);

// tests/Feature/CreateUserTest.php
<?php
namespace Tests\Feature;

use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Support\Facades\Http;
use Tests\TestCase;
use App\Models\User;

class CreateUserTest extends TestCase
{
    use RefreshDatabase;

    public function test_create_user_and_send_welcome_email()
    {
        Http::fake([
            'email.test/*' => Http::response(['ok' => true], 200),
        ]);

        $response = $this->postJson('/api/users', [
            'name' => 'Dan',
            'email' => 'd@ex.com',
        ]);

        $response->assertStatus(200);

        // verify DB
        $this->assertDatabaseHas('users', ['email' => 'd@ex.com']);

        // assert that an outbound call was made
        Http::assertSent(function ($request) {
            return $request->url() == 'http://email.test/send' &&
                   $request['to'] == 'd@ex.com' &&
                   $request['template'] == 'welcome';
        });
    }
}

Tips

• Laravel’s RefreshDatabase will use in-memory sqlite if configured, otherwise migrate a test DB.
• Use Queue::fake() to test that jobs were dispatched without executing background workers.

Practical integration testing strategies

Use transactions for isolation

• Wrap each test in a DB transaction and roll back at the end. Works well when everything runs in the same DB connection.
• Caveat: some ORMs/connections (e.g., tests that spawn separate processes) might not share transaction visibility.

Use in-memory DBs for speed, but test on real DBs in CI

• SQLite in-memory is fast, but can behave differently (indexing, SQL dialect). Complement local tests with real-engine tests in CI using containers.

Stubbing external HTTP reliably

• Python: responses or requests-mock
• Node: nock
• C#: WireMock.Net or replace typed HttpClient with test handler
• PHP: Laravel Http::fake()

Testcontainers — real dependencies in CI

• Spin up a Postgres, Redis, or Kafka container for integration tests.
• Testcontainers exists for many ecosystems (Java, .NET, Node, Python wrappers).

Contract testing for cross-team APIs

• Use Pact or similar to generate contracts from consumer tests and verify provider compliance in provider CI. This avoids brittle mocks and catches breaking API changes.

Background jobs & queues

• Either run job handlers inline (synchronously) in tests, use fake queues that record enqueued messages, or run a worker process in CI that reads from test queue.

How many integration tests should you write?

No fixed number. Aim for:

• Unit tests: many (business logic)
• Integration tests: enough to cover critical integration points (DB persistence, payment flows, auth)
• E2E tests: few (critical user paths)

A practical rule: write integration tests for each major DB operation and for the essential external integrations.

Debugging failing integration tests

• Print request/response bodies and DB rows on failure.
• Capture network traffic (or enable higher logging).
• Reproduce the failing test locally with the same CI container setup (Testcontainers makes that easy).
• If a test is flaky, add instrumentation and increase visibility; temporary retries mask real issues.

Sample integration test checklist

Before merging an integration test into CI:

• Test uses deterministic data.
• DB schema/migrations run and are applied in setup.
• External dependencies are stubbed or provided by test containers.
• Test cleans up (transaction rollback or truncation).
• No sleeps or time-based races.
• Test is focused on behavior, not implementation.

Wrapping up & next steps

Integration tests reduce the mismatch between isolated units and the full system. They give higher confidence than unit tests while being cheaper and faster than full E2E tests. When designed well they catch boundary problems early and make refactoring safer.

Next post in the series: End-to-End (E2E) Testing in Depth — we’ll cover realistic end-to-end strategies, UI-driving vs API-only E2E, test environments, flaky UI tests, and how to design low-maintenance high-value E2E checks.

Happy testing!