Muhammad El-Hindi

Sunday morning reading .. The main takeaway is that BtrLog and the “cache-then-S3” WAL designs used by databases like SlateDB and the object-storage TiKV line of work, look similar at a glance. But they are not the same architecture. The difference is whether local NVMe is a replicated durability layer or only a cache in front of an object storage. Here's what happens with BtrLog when integrated with a database .. • The DBMS talks to a client library, not to S3 • The client assigns LSNs and replicates each append in parallel to a cluster of log nodes • A log node puts the record in an in-memory segment and writes it out-of-place to local NVMe + fsync • The client acks the DBMS only after a quorum of nodes have fsynced Full segments are later archived asynchronously to object storage. That flush is off the commit path. So the hot tail is durable when the quorum acks, even if S3 has not seen the data yet. Local NVMe is there so a power loss or process crash on a log node does not lose the unreplicated-to-S3 tail. Quorum replication is there so losing a log node does not lose committed records. S3 is cheap long-term storage, not the commit device. Now the interesting part is a comparison with SlateDB and classic TiKV with respect to some of the aspects of the durability of appends: 1. Commit Point: • BtrLog - Quorum of log nodes fsynced to local NVMe • SlateDB - WAL SST PUT to object store (or memory if you skip durability) • Classic TiKV - Majority Raft peers fsync local WAL 2. S3 on commit path? • BtrLog - No • SlateDB - Yes, if you want object-store durability • Classic TiKV - No 3. Single-node crash before S3 flush • BtrLog - Survives if quorum remains • SlateDB - Can lose unflushed WAL • Classic TiKV - Survives if Raft majority remains The paper is an interesting read and positions itself quite strongly with so many WAL-on-cloud-storage-with-caching options now available ..