Every unit test passed. The transaction suite was green. I had tests for concurrent writes, for crash recovery, for corrupted logs. By any normal standard, the database was done.
Then I pointed a small verification tool at it, ran a workload that just moves money between bank accounts, and it failed in seconds.
This is the story of the two bugs it found — both invisible to my tests, both the same underlying mistake wearing two different masks — and why “the tests pass” and “the system is correct” are not the same claim.
The database is IgelDB, a small embedded key-value store. But the real subject of this post is the tool that broke it, and the style of testing that makes bugs like these fall out.
Tests check what you thought of. Verification checks what you didn’t.
A unit test is a sentence you write in advance: “when I do X, I expect Y.” It’s only as good as your imagination. If you didn’t think to write down a scenario, the test can’t catch it.
Concurrency bugs live exactly where your imagination runs out. They appear only under specific interleavings of parallel operations — orderings you never explicitly wrote a test for, because there are astronomically many of them.
Verification flips the approach. Instead of asserting specific outcomes, you:
- Generate random concurrent operations from many clients.
- Record the full history of what happened.
- Check that the history obeys a property that must always hold — no matter the interleaving.
You don’t predict the bug. You define what “correct” means, throw chaos at the system, and let the checker find any history that violates correctness. This is the idea behind Jepsen, the well-known distributed-systems testing framework. I wanted that idea in a lighter form for embedded libraries, so I built a scoped-down version — call it Jepsen Lite.
The workload that broke IgelDB is the classic one: a bank.
The bank workload: correctness you can count
Picture a set of bank accounts, each holding a balance. Many clients run concurrently, each doing the same kind of transaction: take some money from account A and move it to account B.
graph LR
subgraph "Many clients, in parallel"
C1["transfer $10<br/>A → B"]
C2["transfer $5<br/>C → A"]
C3["transfer $20<br/>B → D"]
end
C1 --> DB[(IgelDB)]
C2 --> DB
C3 --> DB
Every transfer only moves money — it never creates or destroys it. So there is one property that must hold at every instant, no matter how many transfers run in parallel:
The total of all balances never changes.
That’s the whole trick. The bank workload turns a subtle correctness question into arithmetic. If the total ever drifts, some transfer’s write was silently lost — and the checker just has to add up the numbers.
I ran it. The total drifted. Money vanished.
What the transactions were supposed to guarantee
IgelDB’s transactions use snapshot isolation, the same model most databases use by default. Two rules matter for this story:
1. A transaction reads from a consistent snapshot. When a transaction starts, it sees a frozen picture of the database as of that moment. Other transactions committing afterward don’t disturb its view.
graph TD
T["Transaction starts<br/>takes a snapshot"] --> R1["reads balance A = $100"]
R1 --> R2["reads balance B = $50"]
R2 --> Note["Even if others commit now,<br/>this transaction still sees $100 / $50"]
2. If two transactions write the same key, one must lose. This is what stops lost updates. If two transfers both touch account A at the same time, they can’t both blindly commit — the second one to commit must notice the conflict and be rejected (then retried).
graph TD
subgraph "Correct behavior"
A1["Tx1: change A"] --> A2["Tx1 commits ✓"]
B1["Tx2: change A<br/>same time"] --> B2["Tx2 sees conflict<br/>rejected, retries ✓"]
end
If either rule breaks, transfers can lose money. Both rules broke — in two different places.
Bug #1: the gap between “committed” and “visible”
To detect a write-write conflict, a committing transaction asks: “has anyone else modified the keys I’m about to write, since my snapshot?” If yes, it’s a conflict.
The problem was when a freshly committed write becomes findable by that question.
Committing a transaction in IgelDB happens in two stages:
- Confirm the commit: assign it an order number and write it to the durable log.
- Apply it: make it visible to reads, a moment later, on a background worker.
There’s a gap between those two stages. And the conflict check looked for other transactions’ writes using the read path — which only sees applied writes. So during the gap, a committed-but-not-yet-applied write was invisible to the conflict check.
sequenceDiagram
participant A as Transaction A
participant Log as Durable log
participant Mem as Visible state
participant B as Transaction B
A->>Log: confirm commit (write to A)
Note over Mem: not applied yet
B->>Mem: conflict check: "did anyone write A?"
Mem-->>B: "no" (A's write not visible yet!)
B->>Log: confirm commit (write to A too)
Note over A,B: Both committed. A's change is lost.
Two transfers touch the same account in that window, both are told “no conflict,” both commit. One overwrites the other. Lost update. Money gone.
The fix
The conflict check needed to see commits that were confirmed but not yet applied. So I added a small, temporary record of in-flight commits — writes that are confirmed but haven’t become visible yet. The conflict check now consults both the visible state and this in-flight record.
It’s deliberately tiny: an entry exists only during the gap, and disappears the instant the write becomes visible. It’s not a growing index — it holds only what’s currently in flight.
I re-ran the bank. Conflicts detected went from 0 to 425 — the check was working now. But the total still drifted. 194 lost updates remained.
Something else was wrong.
Bug #2: the same gap, seen from the other side
The first bug was on the write side: a commit was invisible to conflict checks. The second was its mirror image on the read side — and it came from the exact same gap.
Remember rule #1: a transaction reads from a snapshot “as of the moment it started.” I implemented that by pinning the snapshot to the latest confirmed commit number.
But — same gap as before — the latest confirmed commit isn’t the latest applied one. So a transaction would pin its snapshot to a point the visible state hadn’t reached yet. When it then tried to read, the data for that point wasn’t there. It read stale values instead.
sequenceDiagram
participant A as Transaction A
participant State as Visible state
participant B as Transaction B
A->>State: confirm commit #100 (A = $90)
Note over State: only applied up to #95
B->>B: start, pin snapshot = #100
B->>State: read account A as of #100
State-->>B: returns OLD value ($100)<br/>#100 not applied yet
Note over B: B computes on stale data,<br/>overwrites A's $90. Lost update.
A transfer reads an account balance that’s already out of date, does its arithmetic on the wrong number, and writes back a result that erases someone else’s committed change.
The fix
The snapshot must point at something the reader can actually see. So a transaction now pins its snapshot to the latest applied commit — not the latest confirmed one.
This turns out to be not just a fix but correct by definition. IgelDB only tells a client “your commit succeeded” after that commit is applied and visible. So a confirmed-but-not-yet-applied commit hasn’t been acknowledged to anyone — from the outside world’s point of view, it hasn’t committed yet. A new transaction should not see it. Pinning to the applied point is exactly right.
Notice the pleasing asymmetry the two fixes settle into:
- Reads look only at applied, acknowledged commits. They must not see writes that haven’t officially happened.
- Conflict checks look at in-flight commits too. They must still lose to a writer that got there first, even if it isn’t visible yet.
Same gap, two sides, two complementary rules. I re-ran the bank workload. Lost updates: 0. Counter workload: passing. Then I let the tool kill the process mid-write, over and over, and checked that recovery preserved every acknowledged commit. Still zero violations.
Why the tool found what the tests couldn’t
Both bugs needed a very specific alignment to appear: two transactions touching the same key, one landing precisely in the sub-millisecond gap between another’s confirm and apply. I would never have written a unit test for that — I didn’t know the gap existed. That’s the point. You can’t write a test for a bug you can’t imagine.
The verification tool didn’t need to imagine it. It threw thousands of random concurrent transfers at the database and checked one unarguable property — money is conserved. The bug had nowhere to hide, because the checker wasn’t looking for a known failure. It was looking for any history where the invariant broke.
That’s the difference between testing and verification:
- Testing confirms the scenarios you thought of.
- Verification exposes the ones you didn’t.
For most application code, tests are enough. But for the parts of a system where correctness is the whole point — transactions, storage engines, anything concurrent and stateful — “the tests pass” is a weaker claim than it sounds. The interesting bugs are precisely the ones you didn’t think to test.
This is the kind of work Igel Data exists to do: designing, building, and — the part most people skip — verifying data infrastructure. IgelDB and the verification tool in this post are both open source. If your systems have correctness properties that are hard to be sure of, that’s exactly the interesting part.