Storage Engine

VeltrixDB separates keys from values on disk, batches durability writes into group commits, and packs small values at high density — a design built specifically to avoid the write-amplification and compaction stalls inherent to LSM-tree engines like RocksDB.

Why key-value separation

Traditional LSM-tree engines (RocksDB, LevelDB) store full key-value pairs together in sorted SSTables. As data ages, background compaction rewrites the same bytes across levels repeatedly — a value written once may be copied 10-20x over its lifetime before it's finally reclaimed or overwritten. This write amplification consumes NVMe write endurance and steals I/O bandwidth from foreground traffic.

VeltrixDB avoids this with a WiscKey-inspired key-value separation architecture: small, fixed-size key metadata lives in a RAM-resident index, while values live in an append-only on-disk log (the VLog). There is no on-disk B-tree or LSM tree for the index — durability comes entirely from the WAL (Write-Ahead Log), and values are relocated at most once per garbage-collection pass, not repeatedly across compaction levels.

Write-Ahead Log (WAL) and group commit

Each disk has one wal.log file. On unclean shutdown, replayWAL() reads it back and rebuilds the in-memory index; on clean shutdown, wal.truncate() zeroes the file so the next startup skips replay entirely — startup is O(1) rather than O(numLiveKeys).

Every WAL record is a 7-field pipe-delimited text line:

timestamp|isTombstone|key|valueLen|crc32hex|version|vlogOffset[|packed]
[value bytes]

When key-value separation is active, vlogOffset > 0 and no value bytes follow the header — the WAL record is roughly 100 bytes instead of 100 + value_size, because the VLog already holds the durable value bytes.

Group-commit mechanics

Fsync-per-write is the classic throughput killer for any durable log. VeltrixDB amortizes this cost with a group commit pattern:

1. Every Put call enqueues a WALEntry onto a channel.
2. A single flusher goroutine drains up to WALMaxBatchEntries (4096) entries per cycle.
3. One fdatasync call covers the entire batch.
4. All callers waiting on that batch unblock together after the single fdatasync returns.

The default flush window is 10 ms. At 1,000 writes/s the batch is roughly 10 entries; at 100,000 writes/s the batch is roughly 1,000 entries — both pay exactly one fdatasync. This is the single biggest lever for write throughput: N writers sharing one fsync instead of paying N fsyncs turns the durability cost from O(N) into O(1) per window.

Tuning the flush window

The WAL and VLog flush windows (WALFlushWindowMs, VLogFlushWindowMs) should always be kept equal — both default to 10 ms. Lowering the window (e.g. to 5 ms) trades a smaller batch size (lower latency per write) for more frequent fdatasyncs (lower peak throughput); raising it does the opposite. Because the VLog write path is lock-free (see below), pushing the window past ~10 ms yields only marginal batch-size gains — the durability floor, not lock contention, is the binding constraint once the window is wide enough.

Value Log (VLog): append-only design

Each disk has one vlog_active.dat file — a flat, append-only log of value bytes, identified by magic 0x564C5402 ("VLT\x02"). Values are never overwritten in place; a superseded value's space is reclaimed later by the defragmenter (GC).

VLog record format

Offset  Size  Field
  0      4    Magic (0x564C5402)
  4      4    ValLen  (unpadded value byte count)
  8      4    CRC32C  (Castagnoli checksum of value bytes)
 12      4    Reserved (future: compression/schema flags)
 16      8    WriteTimestampUs (int64, µs since epoch)
─────────────────────────────────────────────────────
 24   ValLen  Value bytes (plaintext after encrypt → compress pipeline)
24+V   pad    Zero-padding to next 512-byte sector boundary

Records are sector-aligned to 512 bytes to satisfy O_DIRECT requirements on Linux; on NVMe drives with 4Kn geometry, padding extends to 4096 bytes.

Lock-free offset reservation

The VLog write path avoids a mutex entirely. Offset reservation is a single atomic increment:

offset = vl.end.Add(alignedLen) - alignedLen   // reserves [offset, offset+alignedLen)
pwrite64(fd, value, offset)                     // POSIX-safe concurrent writes

Concurrent pwrite64 calls to non-overlapping byte ranges are race-free per POSIX — no lock is held during the actual I/O. Throughput is therefore bounded only by NVMe IOPS (roughly 450K/disk), not by lock contention. An earlier design held a mutex across the write call; under heavy concurrency that serialized all writers to roughly 1 / WriteAt_latency (~10K writes/s/disk). Removing the mutex was the change that let the engine scale with disk count instead of plateauing.

Block packing: high-density mode

Small values are the worst case for a naive "one record per aligned block" layout: a 128-byte value padded out to a 4096-byte sector wastes over 96% of the block. The VLogBatcher solves this by packing multiple records into a single 4 KB block on the batched write path (MultiPut, GC relocations, the write batcher) — the single-Put hot path stays on the simpler unpacked fast path.

For 128-byte values: header (24 B) + value (128 B) = 152 B per record, so a 4096-byte block holds up to 26 records — a 27x density improvement over one record per 4 KB block.

• FlagPacked (0x40) is set on the IndexEntry for packed records.
• MarkDead(valueLen) subtracts only header + value bytes (not the full 4 KB block) when FlagPacked is set — this keeps the garbage-ratio accounting accurate under packing.
• The read path is unchanged: ReadValue(DiskOffset, ValueSize) rounds the offset down to the nearest 4 KB block boundary and extracts the record at offset % 4096. Packed records simply have non-4K-aligned DiskOffset values (e.g. blockOff + 152, blockOff + 304, …).

Oversized records (header + value larger than one block) automatically fall back to the unpacked path inside the batcher, so packing never has to special-case large values.

LIRS cache

VeltrixDB uses LIRS (Low Inter-Reference Recency Set) rather than plain LRU for its in-memory value cache. LIRS is scan-resistant: a sequential scan over cold data does not evict the hot working set, which is a well-known failure mode of LRU under mixed scan/point-lookup workloads.

• Small values (≤ 256 bytes) are assigned priority = 2 — scan-resistant and harder to evict.
• Large values are assigned priority = 1.
• A 16-entry scan window is used to pick the largest cold victim for eviction.
• Cache size is configurable via -cache <MB>; production deployments typically size it at 256 GB or larger.

The priority split matters operationally: high-cardinality small-value workloads (e.g. counters, session flags, feature lookups) are exactly the pattern most vulnerable to premature eviction under LRU, and the priority-2 treatment keeps them resident even when a large scan (backup, range query, GC read pass) runs concurrently.

VLog garbage collection: three-tier escalation

Dead VLog space accumulates from overwrites and deletes. The Defragmenter reclaims it by walking live index entries, rewriting live records to a fresh VLog position, updating IndexEntry.DiskOffset via CAS (safe under concurrent reads), calling MarkDead(oldValueLen) to account for freed bytes, and issuing fallocate(PUNCH_HOLE) or BLKDISCARD (on raw NVMe) to return dead pages to the OS.

GC bandwidth and pacing escalate automatically as garbage accumulates, specifically to avoid a death spiral: high read latency pauses GC, which lets garbage keep growing, which increases cache misses and disk seeks, which keeps read latency high — permanently suppressing GC. The three-tier design exists to break that cycle before it becomes irreversible.

The emergency tier is the escape hatch: once garbage crosses 65%, GC bypasses the admission-control pause entirely, runs at uncapped bandwidth, and checks in on a quartered interval. The system logs this loudly and increments a dedicated counter so operators can see it happening:

[gc] disk=N EMERGENCY  garbage=67.2%  ewma=21.4ms  bypassing admission-control pause to escape death spiral

Metric: veltrixdb_vlog_gc_emergency_runs_total. A persistent non-zero rate on this counter is the canonical production signal that sustained write rate is outpacing GC reclaim throughput — the fix is to investigate disk IOPS headroom or reduce write load, not to disable the emergency tier.

For how the read-latency EWMA interacts with these tiers, and the full admission-control ladder, see Admission Control & GC.

On-disk layout (per disk)

/data-dir-N/
├── wal.log              — Write-Ahead Log (group-commit, 7-field text)
├── vlog_active.dat      — Value Log (24-byte header + sector-aligned values)
├── seg_XXXXXXXX.dat     — Segment files (O_DIRECT sequential, 64-byte header)
├── vlog_punch_watermark — GC punch offset watermark for defragmentation
└── raft_state.gob       — Raft persistent state (term, votedFor, log entries)

With multiple disks (-data-dirs /mnt/nvme0,...,/mnt/nvme7), each disk gets its own independent WAL, VLog, segment files, and compaction goroutine. Shard i always lives on disk i % numDisks — a failure on one disk does not affect the other disks' shards.

Value transform pipeline

Write path:   plaintext  →  compress  →  encrypt  →  VLog bytes on disk
Read  path:   disk bytes →  decrypt   →  decompress →  plaintext

Compression runs before encryption: encrypted ciphertext has high entropy and is effectively incompressible, so compressing after encryption would waste CPU for no space savings. FlagCompressed and FlagEncrypted are stored per-record on the IndexEntry, so records written before encryption was enabled remain readable — the flags, not a global setting, decide which transforms the read path applies.

Compression uses zstd and is enabled for values ≥ 256 bytes. Encryption uses AES-256-GCM with a unique 12-byte nonce per record, keyed from the VELTRIXDB_ENCRYPTION_KEY environment variable (base64) or a --encryption-key-path file.

Crash recovery

On unclean shutdown (crash, OOM kill, SIGKILL):

1. replayWAL() opens wal.log on each disk.
2. Each record is parsed against the 7-field format and its CRC32C is verified.
3. applyWALReplay() rebuilds the in-memory sharded index.
4. For KV-separated records (vlogOffset > 0), the VLog already holds the value bytes — the WAL entry only re-establishes the index pointer, without re-reading the value.
5. Legacy 6-field entries (no vlogOffset) are still supported for backward compatibility.

If a process crashes after the VLog write but before the WAL write completes, the value bytes are orphaned but were never referenced by the index — they are simply reclaimable garbage on next GC, and the client never received an acknowledgment. This ordering (VLog write, then WAL write) is what keeps crash recovery safe without requiring the two writes to be atomic with each other.