Jepsen Lite just hit v0.2.0. This time the subject isn’t IgelDB — it’s two real, widely-used embedded stores: SQLite and LMDB. Both now live under cases/ as complete, runnable verification suites, and they exist to answer one question: when you configure a store to be durable, is it actually durable?

The fault everyone reaches for isn’t enough

The obvious way to test a store’s durability is to kill it. Jepsen Lite has always been able to do this — SIGKILL the process mid-run, restart it, and check whether everything it had acknowledged is still there.

That test is worth running, and both suites run it. But it proves less than it looks like it does, for a simple reason: SIGKILL kills the process, not the kernel. Any acknowledged write the store handed to the operating system is still sitting in the OS page cache, and the OS keeps flushing that cache to disk on its own schedule, crash or no crash. So, for the acknowledged writes these suites check, a store configured to fsync on every commit and a store configured to skip fsync entirely come through a kill -9 test identically. No amount of repetition changes that — the page cache remains outside the killed process.

That gap matters because fsync (or its absence) is usually the actual durability knob people flip for performance:

A verification tool that can’t tell these settings apart isn’t really testing durability at all — it’s testing “does the process restart okay,” which is a different and much easier question.

power-off: the fault that actually asks

Jepsen Lite’s answer is a fault it calls :power-off, built on lazyfs. Instead of trusting the OS page cache, Jepsen Lite mounts the target’s data directory on a FUSE filesystem that holds writes in a cache of its own until the target explicitly calls fsync. A power-off:

  1. tells lazyfs to clear its cache — dropping precisely the bytes that were never fsynced,
  2. waits for that to be confirmed,
  3. then sends SIGKILL.

What restarts afterwards is a store recovering from a disk that genuinely lost whatever it never synced. That’s the experiment kill -9 can’t run.

Two honesty notes worth keeping in mind before the numbers below make it look more magical than it is:

  • A failing power-off run is evidence of a store bug only if the store was configured to fsync. A store deliberately configured not to sync should lose data on power-off — that’s the fault injector working, not a bug.
  • It models the filesystem, not the disk. lazyfs drops what was never fsynced; it doesn’t simulate a drive’s own write cache, and it doesn’t explore write reordering the way a real crash could. It’s one honest, deterministic slice of “what unsynced data looks like,” not a full crash-consistency oracle.
  • It currently runs only on Linux, with FUSE, for a local-process target. Jepsen Lite refuses the fault when those prerequisites are missing rather than silently downgrading it to a plain process kill.

SQLite: the difference is total

The SQLite case runs set — a workload where every acknowledged write must survive — under three conditions, at two synchronous settings:

  kill -9 power-off
synchronous=FULL 0 lost 0 lost
synchronous=OFF 0 lost 1238 of 1598 lost

Read that bottom-left cell again: a database that’s been told don’t bother syncing sails through a kill -9 test with zero losses. Only the power-off column can tell FULL and OFF apart — and once it can, the difference is enormous: three quarters of the acknowledged writes are gone.

Since SQLite in WAL mode also checkpoints on a clean close, the suite runs a third mode too — a clean close/open cycle rather than a hard kill — and that, unsurprisingly, is even further from testing fsync: closing flushes and checkpoints on purpose, so it tells you even less about unsynced data than a kill does.

LMDB: the failure mode is worse than “lost data”

LMDB’s story has an extra twist. LMDB is copy-on-write with two meta pages, and normally a crash needs no recovery at all — the last valid meta page just is the last committed transaction, so a kill -9 test looks almost too easy to pass:

  kill -9 power-off
synced (default) 0 lost 0 lost
MDB_NOSYNC 0 lost corrupted, or writes lost

With MDB_NOSYNC, four out of five power-off runs came back with MDB_INVALID — the environment couldn’t even be reopened, because lazyfs’s cache-clear doesn’t preserve write order, and it managed to leave a meta page pointing at data pages that never made it to disk. The fifth run did reopen, having lost 1,506 of 1,518 acknowledged writes. Meanwhile kill -9 against the exact same MDB_NOSYNC config lost nothing at all — and ran about 78× faster, which is exactly the trap: it would be easy to read that speed-up plus a clean kill run as “MDB_NOSYNC is safe,” and it would be wrong. This is precisely what LMDB’s own documentation says MDB_NOSYNC risks — the fault just makes it observable instead of theoretical.

Why this is the case worth publishing

The point of these two cases isn’t “SQLite and LMDB are unreliable” — configured to sync, both come through every fault (crash, kill, power-off) with nothing lost. The point is that a fault that can’t distinguish “syncs” from “doesn’t sync” can’t be trusted to validate a durability claim either way — and a fault that fails exactly when a store is misconfigured, and passes exactly when it isn’t, is the evidence that power-off is measuring the thing it claims to measure. That’s also why both READMEs are careful to say what power-off doesn’t cover — disk write caches, write reordering — rather than let one green run stand in for durability in general.

Both are real, independently-runnable projects under cases/sqlite and cases/lmdb, each with its own deps.edn, its own driver process for kill -9, and a full README with the runnable commands and the honest limits. If you maintain an embedded store and have ever wondered whether your durability setting actually does anything under a crash, cases/ is the fastest way I know to find out.