Key-value separation & why compaction shouldn't touch values.

Compaction is the silent tax you pay on every read.

Classical LSM trees — the storage engine inside RocksDB, Cassandra, LevelDB, and a dozen others — rewrite your values during compaction. That rewrite is a large part of their write amplification, and of the latency spikes compaction can cause.

Here's the shape of the problem. An LSM tree absorbs writes into a fast in-memory level (the memtable), then flushes that level to disk as an immutable SSTable. To keep read amplification bounded, the engine periodically merges SSTables — that's compaction. And compaction has to rewrite every byte that lives in the SSTables being merged: keys and values.

If your values are 128 bytes and your keys are 16 bytes, you've just paid an 8× rewrite tax on data that didn't change. Measured end-to-end across the entire lifecycle of a value, the industry term for this is write amplification. Commonly cited ranges for LSM engines are in the high single digits or more; we have not measured them ourselves.

VeltrixDB makes one architectural decision to avoid this: values never go into an index structure that gets compacted. Values go into an append-only value log (VLog). The index is an in-RAM hash table of 64-byte metadata entries with no value bytes, and it is made durable by the WAL rather than by compaction.

We have not published a write-amplification measurement. An earlier version of this post claimed a measured 1.0×; that was never measured, and ~1× is a design property of batched (packed) writes, not a benchmark result. VeltrixDB still writes more than the client sends: a WAL record per write (header and key, no value bytes), 4 KB block alignment — a single small Put takes a whole 4 KB block — and VLog GC, which copies live values forward. The measured numbers we do have are on the performance page.

WiscKey, production-hardened.

In 2016, a paper out of Wisconsin proposed the separation we use today. We took the idea, kept its core insight, and rebuilt the engine end-to-end to make it survive in multi-cloud production.

The WiscKey paper (FAST '16) made one elegant observation: if you separate keys from values, only the keys have to participate in compaction. Values can live in an append-only log on disk — they're written once and only touched again when garbage collection copies the still-live ones out of a mostly-dead region.

The original WiscKey paper proved it worked in a research setting. VeltrixDB takes the idea further — we re-architected it around three production realities the paper sidestepped:

• Sharding and multiple disks · the index is split into 8192 shards (low 13 bits of FNV-1a), each with its own RWMutex and Bloom filter. Each NVMe disk has one WAL and one VLog, and shard i lives on disk i % numDisks
• Group commit · concurrent writers share one WAL write(2) and one fdatasync, a lone writer is synced at once (adaptive pacing, v1.12), and a batch goes to the VLog as one pwrite plus one fdatasync. (An io_uring VLog write bridge exists but is opt-in via VELTRIXDB_URING_BRIDGE and off by default.)
• Caching · the paper had a basic LRU; we replaced it with LIRS, which dramatically improves hit rate under skewed access

What lives in the index

Each VeltrixDB shard keeps a hash table of its keys, entirely in RAM. On cgo builds (the default, macOS included) it is an off-heap C++ table the Go GC never scans. CGO_ENABLED=0 builds use a Go map. Entries don't store values. Each one is a 64-byte IndexEntry recording where the value sits in the VLog, its on-disk and plaintext sizes, its checksum, TTL and flags. (A C++ ART index exists under cpp/, but nothing in Go calls it.)

The whole live key space has to fit in RAM, and that is why the fit check is upfront about the RAM budget. The figure we have measured: with 5M keys, the native index settled at ~142 bytes of RSS per key, including key bytes, plus ~90 B per key for the ordered index that serves RANGE / SCANCUR unless you start with --disable-ordered-index. So the RAM saving over an all-in-RAM store depends on value size: with 128 B values the index is about as large as the values; with 1 KB values it is roughly 4–7× less RAM. Numbers for larger key counts would be extrapolations; we have not measured them. The payoff is that no index compaction ever rewrites value bytes.

Value lands on NVMe before ack.

A write isn't acknowledged until the value is durable on disk. The VLog append and the WAL append run concurrently, each ending in an fdatasync, and the caller waits for the slower of the two, not their sum.

When a write enters VeltrixDB it follows these stages. We have not published a per-stage latency breakdown, so none is shown:

Group-commit WAL

There is one WAL per disk. Writers enqueue their record on a channel. A single flusher goroutine drains up to 4096 records per cycle, writes the whole batch with one write(2), and covers it with one fdatasync. Then every waiting writer is released. The window is 15 ms (--wal-flush-window-ms, which must match --vlog-flush-window-ms).

Since v1.12 the pacing is adaptive by default (--group-commit=adaptive): a writer that is alone is synced immediately, and concurrent writers are flushed after an idle gap of about one fdatasync, so the window is only an upper bound. --group-commit=fixed restores the full-window wait. Measured with an emulated 300 µs fdatasync on macOS: a lone writer's P50 went from 16 ms to 0.45 ms, and 64 writers from 4.0K to 37.5K writes/s at the same 64 writes per sync.

Either way the fdatasync cost is amortized across every concurrent writer in the batch, and no caller is answered before the sync that covers its bytes. Records are binary: a 48-byte header, then the key, then a CRC32C over the whole record. So keys containing | or newlines are safe. The binary WAL format shipped in v1.6 and the single-write(2) flusher in v1.5.

VLog append

The value goes to the VLog, a single append-only file per disk (vlog_active.dat). There is no mutex: a writer reserves its offset with an atomic add and issues a pwrite. On Linux the file is opened with O_DIRECT, and every record starts on a 4 KB boundary — a single Put pads its record to a whole 4 KB block. A MultiPut batch (and GC relocation) instead packs small values into shared 4 KB blocks — up to ~26 records of 128 B (24 B header + value) per block — and writes the whole extent with one pwrite and one fdatasync.

Values are never overwritten in place. Space from overwrites and deletes is reclaimed later by the defragmenter, which copies live records forward and punches holes (fallocate(PUNCH_HOLE), or BLKDISCARD on raw block devices).

Index update

Finally, the IndexEntry is installed in the shard's table under that shard's RWMutex. There are 8192 shards, so writes to different shards don't contend. If the process crashes between the VLog write and the WAL sync, the offset never reaches the WAL: the client never got an OK, and the orphaned bytes are reclaimed by GC.

Cache, Bloom filter, index — then one pread.

Most reads should never reach the disk. The ones that do cost one aligned pread from the VLog, checked by checksum.

A GET hashes the key once and uses that hash for both the cache and the index. First comes the sharded LIRS cache: a hit returns in about 92 ns in-process. On a miss, the shard's Bloom filter answers "definitely absent" without taking a lock or allocating. Next is the shard's index entry, which covers absent, tombstoned and TTL-expired keys. Only then does the read touch the VLog.

The disk read: Go pread

The VLog read is plain Go (storage/vlog.go, ReadValue). It issues one pread, aligned down to the 4 KB block, and extracts the record at its offset within the block. It verifies the record magic and CRC32C, then decrypts and decompresses as needed, and inserts the value into the cache. The read path does not use io_uring and is not zero-copy. An SQPOLL-based C++ reader is compiled into cgo builds, but nothing in Go calls it.

Since v1.7, GetNoIO runs the same checks but stops before the pread. It reports that I/O is needed, so the opt-in C++ network front-end can hand disk reads to a goroutine instead of blocking its event loop.

O_DIRECT — out of the page cache

On Linux, VeltrixDB opens the VLog with O_DIRECT, so the kernel bypasses the page cache. Other platforms use buffered I/O. Two consequences on Linux:

• Deterministic latency · no page cache evictions, no surprise jitter from a noisy tenant doing a large sequential scan
• Less RAM pressure · we choose what to cache; the kernel doesn't fight us by caching VLog blocks we'll never re-read

8192 shards, N disks

The index is split into 8192 shards, each with its own RWMutex and Bloom filter, so concurrent reads on different shards don't contend. The cache is split separately into up to 256 shards, keyed on the high hash bits. Disk reads spread across every NVMe disk you give the server with --data-dirs, because shard i lives on disk i % numDisks.

LIRS — because LRU isn't smart enough.

The cache's job is to keep the hot set resident even when something scans cold data. The way to do that isn't more cache — it's a better eviction policy than LRU.

Most databases use LRU (Least Recently Used) for their block cache. LRU has one famous failure mode: a large cold scan walks through every block once, evicting the hot working set on its way through. Your "hot" cache is now full of cold data you'll never read again.

VeltrixDB uses LIRS (Low Inter-reference Recency Set). LIRS keeps two stacks — one for LIR blocks (proven hot) and one for HIR blocks (probationary). A block has to demonstrate reuse before it gets promoted out of probation, which means a single-pass scan can't pollute the resident set.

Why this matters in production

Consider a session store: 90% of requests hit the last hour of sessions; 10% are background jobs reading older history. With LRU, the background jobs would constantly evict hot sessions and your cache hit rate would collapse to ~60%. With LIRS, the background blocks live in HIR, get evicted on their second non-use, and the hot set stays resident. (The 60% and hit-rate figures here are illustrative, not measured. A 94.2% hit rate from an internal 1B-key GKE run was quoted here earlier; that run is unverified. The one reproducible read run served its whole dataset from cache.) The cache is sized with -cache in MB. The default is 256 MB, and production deployments size it far larger.

GC backs off when reads slow down.

Tombstones, deletes, and overwrites leave dead values in the VLog. We reclaim that space, and the engine throttles or pauses that work automatically when read latency rises.

The defragmenter walks live index entries and copies live records forward in the VLog, oldest first, skipping keys written in the last 30 s because they are likely to be overwritten again. It updates each entry's offset with a CAS, which is safe under concurrent reads, then punches holes to return the dead pages. How hard it works escalates with the garbage ratio. This is automatic, not a per-shard mode:

• < 30% garbage · no GC
• 30–50% · 60 MB/s cap while reads are slow (uncapped otherwise), respects the admission-control pause, 120 s defrag interval
• 50–65% (critical) · 200 MB/s cap while reads are slow, still respects the pause, 60 s interval
• ≥ 65% (emergency) · uncapped, ignores the pause so GC can't be starved into a death spiral, 30 s interval

What protects reads is admission control driven by a read-latency EWMA. GC and reads share the same disks and I/O path; there is no separate io_uring instance or cgroup weighting. At a read EWMA of 15 ms or more, GC bandwidth is capped at 60 MB/s. Above 20 ms, GC pauses and each PUT sleeps 2 ms. Everything resumes once the EWMA falls below 10 ms, and a pause left over from a write-only period clears by itself once no read has arrived for 4 minutes.

Four decisions, one outcome.

No single trick makes a disk-backed store predictable. It's the compound effect of four design decisions. (An earlier version of this paragraph quoted 4.9 ms P99 on 1 billion keys. That figure comes from an internal GKE run and is unverified.)

Each decision is independently boring:

• Key-value separation · the index holds 64-byte metadata entries in RAM; values are written once to the VLog and moved only by GC
• Group commit + one aligned pread · one write(2) and one fdatasync per WAL batch, and one checksummed pread per disk read (O_DIRECT on Linux)
• Sharded LIRS cache · cold scans have a hard time evicting the hot set
• Admission-controlled GC · cleanup is capped or paused when read latency rises

That's the entire engineering bet. Whether it's worth it for your workload depends on whether you live in the failure mode the design targets: latency spikes from compaction rewriting values. The benchmark post has the measured numbers and the exact conditions they were taken under.