Storage Engine

VeltrixDB separates keys from values on disk, batches durability writes into group commits, and packs small values at high density — a design built specifically to avoid the write-amplification and compaction stalls inherent to LSM-tree engines like RocksDB.

Why key-value separation

Traditional LSM-tree engines (RocksDB, LevelDB) store full key-value pairs together in sorted SSTables. As data ages, background compaction rewrites the same bytes across levels repeatedly — a value written once may be copied 10-20x over its lifetime before it's finally reclaimed or overwritten. This write amplification consumes NVMe write endurance and steals I/O bandwidth from foreground traffic.

VeltrixDB avoids this with a WiscKey-inspired key-value separation architecture: small, fixed-size key metadata lives in a RAM-resident index, while values live in an append-only on-disk log (the VLog). There is no on-disk B-tree or LSM tree for the index — durability comes entirely from the WAL (Write-Ahead Log), and values are relocated at most once per garbage-collection pass, not repeatedly across compaction levels.

In-memory index

The index is split into 8192 shards (FNV-1a(key) & 0x1FFF), each with its own RWMutex and a lock-free Bloom filter. Each shard holds a key → 64-byte IndexEntry table; the full live key space is always in memory. The table implementation depends on the build:

VELTRIXDB_INDEX=map opts a cgo build out of the native table (=native forces it); VELTRIXDB_DISABLE_CGO_ENGINE=1 also selects the map, and so does a toolchain older than Go 1.21 (the native table is built only with cgo && go1.21). The server logs the choice at startup: [index] implementation=native or =map.

IndexEntry (64 bytes, one cache line)

storage/types.go checks the size at init. No value bytes are stored; the engine's version counter goes into the WAL record, not the entry.

A separate ordered skiplist (~90 B/key) serves RANGE and SCANCUR; --disable-ordered-index turns it off. A key is inserted into it only when it becomes live (absent or tombstoned before) — overwriting a live key does not touch it.

Write-Ahead Log (WAL) and group commit

Each disk has one wal.log file. On unclean shutdown, replayWAL() reads it back and rebuilds the in-memory index. On clean shutdown, the engine does not truncate the WAL: it writes a compacted checkpoint (one record per live key, tombstones dropped) to wal.log.ckpt, fdatasyncs it, and atomically renames it over wal.log. The next startup replays O(live keys) records instead of the full write history, and a crash mid-checkpoint leaves the old WAL intact.

New in v1.6 New records are binary (storage/wal_format.go), all integers little-endian: a 48-byte header, then the key, then the value (only when it is stored inline), then a CRC32C over the whole record.

Offset  Size  Field
  0      1    Magic 0xB1 (a text record always starts with an ASCII digit)
  1      1    Format version (1)
  2      2    Flags: bit0 tombstone, bit1 packed, bit2 value inline
  4      4    Key length
  8      4    valueLen  (plaintext length)
 12      4    diskLen   (on-disk length after compress + encrypt)
 16      4    CRC32C of the plaintext value
 20      1    xflags: FlagCompressed | FlagEncrypted
 21      3    Zero
 24      8    Timestamp (UnixNano)
 32      8    Version
 40      8    vlogOffset (0 = value inline, or none)
─────────────────────────────────────────────────────
 48    keyLen  Key bytes
  …    valueLen Value bytes (only when the inline flag is set)
 end     4    CRC32C of every byte above

Records are length-prefixed, so keys containing | or \n are safe. With the old text format such a key made replay stop at that record and drop every later acknowledged write on a crash restart, and the text header had no checksum.

When key-value separation is active, vlogOffset > 0 and no value bytes follow — the WAL record is the 48-byte header plus the key plus the 4-byte CRC, because the VLog already holds the durable value bytes.

Replay still reads legacy text records, in any mix with binary ones (the first byte of each record decides), so an upgraded node keeps appending binary records after existing text ones:

timestamp|isTombstone|key|valueLen|crc32hex|version|vlogOffset|packed|diskLen|xflags
[value bytes]

--wal-format=text (server only; it logs a warning) writes the legacy format and exists only to roll back to a pre-binary build: run once with it, stop cleanly so the checkpoint rewrites wal.log as text, then downgrade. Fixed in v1.14 a key written with a TTL gets a version-2 record (the same header with byte 1 = 2, then its 8-byte absolute expiry in Unix µs) or an 11th text field, so TTLs survive restarts (in v1.13 and earlier the WAL carries no TTL and TTL’d keys come back without one); TTL-free records stay version 1, byte for byte. v1.13 and older cannot read those records, so the rollback mode to them becomes --wal-format=text-legacy; v1.14 and later (v1.16 included) read them directly and need no rollback step. veltrixdb-backup, PITR restore and veltrix-repair always write binary.

Group-commit mechanics

Fsync-per-write is the classic throughput killer for any durable log. VeltrixDB amortizes this cost with a group commit pattern:

1. Every Put call enqueues a WALEntry onto a channel; a MultiPut serializes all its records into one buffer and enqueues it with a single channel send.
2. A single flusher goroutine drains up to WALMaxBatchEntries (4096) records per cycle.
3. The whole batch is written with one write(2), and one fdatasync call covers it.
4. All callers waiting on that batch unblock together after the single fdatasync returns.

When a batch is flushed (storage/group_commit.go, --group-commit):

• adaptive (default) New in v1.12 — the flush window (15 ms, --wal-flush-window-ms) is only an upper bound. A writer that is alone (the batch holds one request and recent batches held about one) is synced immediately. Otherwise the batch stays open until no request has arrived for an idle gap of about one fdatasync (an EWMA of measured sync time, clamped to 20 µs–2 ms), the window expires, or 4096 records are queued (--wal-max-batch). Requests that arrive during a sync form the next batch.
• fixed — every batch waits the whole window: at 1,000 writes/s roughly 15 entries per batch, at 100,000 writes/s roughly 1,500, each paying one fdatasync.

The VLog flusher uses the same pacer. Durability is identical in both modes: no caller is answered before the fdatasync that covers its bytes. Measured on macOS with an emulated 300 µs sync: a lone writer's P50 was 0.45 ms adaptive vs 16 ms fixed; 64 writers reached 37.5K vs 4.0K writes/s at the same 64 writes per fdatasync.

Tuning the flush window

The WAL and VLog flush windows (WALFlushWindowMs / --wal-flush-window-ms, VLogFlushWindowMs / --vlog-flush-window-ms) should always be kept equal — both default to 15 ms. In fixed mode, lowering the window (e.g. to 5 ms) trades a smaller batch size (lower latency per write) for more frequent fdatasyncs (lower peak throughput); raising it does the opposite. In adaptive mode the window only caps how long a busy batch may stay open. Because the VLog write path is lock-free (see below), widening the window well past the default yields only marginal batch-size gains — the durability floor, not lock contention, is the binding constraint once the window is wide enough.

Value Log (VLog): append-only design

Each disk has one vlog_active.dat file — a flat, append-only log of value bytes, identified by magic 0x564C5402 ("VLT\x02"). Values are never overwritten in place; a superseded value's space is reclaimed later by the defragmenter (GC).

VLog record format

Offset  Size  Field
  0      4    Magic (0x564C5402)
  4      4    ValLen  (unpadded value byte count)
  8      4    CRC32C  (Castagnoli checksum of value bytes)
 12      4    Reserved (future: compression/schema flags)
 16      8    WriteTimestampUs (int64, µs since epoch)
─────────────────────────────────────────────────────
 24   ValLen  Value bytes as stored (output of compress → encrypt)
24+V   pad    Zero-padding to the next 4096-byte block boundary

Every record offset is 4 KB-aligned (vlogBlockSize = 4096), which is O_DIRECT-safe on both XFS and 4Kn NVMe drives. A single Put pads its record to a whole 4 KB block. The CRC32C here covers the stored bytes; IndexEntry.CRC32C is the checksum of the plaintext. With --raw-vlogs (Linux) the VLog is a raw NVMe block device with a 4 KB superblock at offset 0.

Lock-free offset reservation

The VLog write path avoids a mutex entirely. Offset reservation is a single atomic increment:

offset = vl.end.Add(alignedLen) - alignedLen   // reserves [offset, offset+alignedLen)
pwrite64(fd, value, offset)                     // POSIX-safe concurrent writes

Concurrent pwrite64 calls to non-overlapping byte ranges are race-free per POSIX — no lock is held during the actual I/O. Throughput is therefore bounded only by NVMe IOPS (roughly 450K/disk), not by lock contention. An earlier design held a mutex across the write call; under heavy concurrency that serialized all writers to roughly 1 / WriteAt_latency (~10K writes/s/disk). Removing the mutex was the change that let the engine scale with disk count instead of plateauing.

Batch writes (MultiPut) go through the VLogBatcher, which reserves the whole batch's extent with one atomic add, stages the records into one contiguous 4 KB-aligned buffer, and writes it with one pwrite plus one fdatasync per disk, however many blocks it spans.

New in v1.7 On Linux cgo builds the batch write can instead be submitted through an io_uring bridge (one ring per disk). It is off by default, since the batch is already a single pwrite. Opt in with VELTRIXDB_URING_BRIDGE=on (no SQPOLL) or sqpoll; VELTRIXDB_DISABLE_CGO_ENGINE=1 forces it off. If ring setup fails (for example, the Kubernetes RuntimeDefault seccomp profile blocks io_uring_setup) the engine logs it and falls back to pwrite.

Read path

Get checks, in order: the LIRS cache, the index (absent / tombstone / expired), a dirty value still in RAM, and finally the VLog. On v1.13 and earlier a cache hit is returned without an expiry check, so a cached key can be served after its TTL. Fixed in v1.14 each cache entry carries its key’s TTL deadline, and a hit past it falls through to the index (tombstone + evict + not-found); immortal keys pay one compare. The VLog read is Go code (storage/vlog.go ReadValue): a 4 KB-aligned pread that verifies the record magic and CRC32C. It is not io_uring and not zero-copy.

New in v1.7 GetNoIO runs the same steps but stops before the VLog read and returns needIO=true instead; GetAfterNoIO completes that key, and the pair counts as one read. It exists for callers that must not block on disk, such as the opt-in C++ network front-end's event loops.

Block packing: high-density mode

Small values are the worst case for a naive "one record per aligned block" layout: a 128-byte value padded out to a 4096-byte sector wastes over 96% of the block. The VLogBatcher solves this by packing multiple records into a single 4 KB block on the batched write paths — MultiPut and GC relocation, the only two callers of VLogBatcher. A single Put stays on the unpacked path, one block per record.

For 128-byte values: header (24 B) + value (128 B) = 152 B per record, so a 4096-byte block holds up to 26 records — roughly 26× denser than one record per 4 KB block.

• FlagPacked (0x40) is set on the IndexEntry for packed records.
• MarkDead(valueLen, packed) subtracts only header + value bytes (not the full 4 KB block) when packed is true (callers pass entry.IsPacked()) — this keeps the garbage-ratio accounting accurate under packing.
• The read path is unchanged: ReadValue(DiskOffset, ValueSize) rounds the offset down to the nearest 4 KB block boundary and extracts the record at offset % 4096. Packed records simply have non-4K-aligned DiskOffset values (e.g. blockOff + 152, blockOff + 304, …).

Oversized records (header + value larger than one block) automatically fall back to the unpacked path inside the batcher, so packing never has to special-case large values.

LIRS cache

VeltrixDB uses LIRS (Low Inter-Reference Recency Set) rather than plain LRU for its in-memory value cache. LIRS is scan-resistant: a sequential scan over cold data does not evict the hot working set, which is a well-known failure mode of LRU under mixed scan/point-lookup workloads.

• Small values (≤ 256 bytes) are assigned priority = 2 — scan-resistant and harder to evict.
• Large values are assigned priority = 1.
• A 16-entry scan window picks the victim with the highest size ÷ priority.
• The cache is split into up to 256 independent LIRS shards, each with its own lock and a slice of the budget, selected by the high bits of the key hash (the index uses the low 13). A single cache-wide mutex had put 21.7% of CPU in one lock; sharding took a cache hit from 711 ns to 92 ns. The trade-off: per-shard budgets are not one global budget, so a skewed keyspace can evict from a hot shard while a cold one has room.
• Default size 256 MB (-cache <MB>; --read-heavy presets 400 GB).

The priority split matters operationally: high-cardinality small-value workloads (e.g. counters, session flags, feature lookups) are exactly the pattern most vulnerable to premature eviction under LRU, and the priority-2 treatment keeps them resident even when a large scan (backup, range query, GC read pass) runs concurrently.

VLog garbage collection: three-tier escalation

Dead VLog space accumulates from overwrites and deletes.

The Defragmenter (storage/defrag.go) wakes every DefragInterval (120 s; 300 s with --read-heavy) and compacts every disk whose garbage ratio is at least --gc-threshold (default 0.30). It snapshots the VLog end as the horizon, skips keys written in the last 30 s, restages live records oldest-first through a VLogBatcher (256 records per pwrite + fdatasync), moves IndexEntry.DiskOffset by CAS (a concurrent Put that wins leaves the new copy as garbage; if more than 20% of a batch loses its CAS, each Put sleeps 1 ms for 200 ms), marks the old copy dead, and punches out the dead head of the file with fallocate(PUNCH_HOLE) or BLKDISCARD on a raw device. Every pass ends by rebuilding the bloom filters.

GC bandwidth and pacing escalate automatically as garbage accumulates, specifically to avoid a death spiral: high read latency pauses GC, which lets garbage keep growing, which increases cache misses and disk seeks, which keeps read latency high — permanently suppressing GC. The three-tier design exists to break that cycle before it becomes irreversible.

Bandwidth reads "unlimited while the read EWMA ≤ 15 ms / capped above it": the caps apply only when reads are slow. Intervals are DefragInterval, halved and quartered by the worst disk's ratio.

The emergency tier is the escape hatch: once garbage crosses 65%, GC bypasses the admission-control pause entirely, runs at uncapped bandwidth, and checks in on a quartered interval. The system logs this loudly and increments a dedicated counter so operators can see it happening:

[gc] disk=N EMERGENCY  garbage=67.2% ≥ 65%  ewma=21.4ms  bypassing admission-control pause to escape death spiral

Metric: veltrixdb_vlog_gc_emergency_runs_total. A persistent non-zero rate on this counter is the canonical production signal that sustained write rate is outpacing GC reclaim throughput — the fix is to investigate disk IOPS headroom or reduce write load, not to disable the emergency tier.

For how the read-latency EWMA interacts with these tiers, and the full admission-control ladder, see Admission Control & GC.

On-disk layout (per disk)

/data-dir-N/
├── wal.log          — Write-Ahead Log (group-commit, binary records)
├── wal.log.ckpt     — transient: checkpoint being written
├── vlog_active.dat  — Value Log (24 B header, 4 KB-aligned)
├── seg_active.dat   — Segment file (KV separation off only)
└── .vlog_punch_wm   — GC punch-hole watermark (8 B offset)

The first data dir also holds index_defs.json (secondary-index definitions). Raft state is not per disk: in --mode=raft it lives in <--data>/raft/. On v1.14 and earlier that is raft_state.gob (rewritten whole on every commit) and raft_snapshot.gob. Fixed in v1.15 the log is the append-only, CRC-checked raft_log.dat, term and vote are in raft_meta.dat, and raft_snapshot.gob is unchanged; an existing raft_state.gob is migrated on first start. Fixed on main after v1.16 — ships in the next release the snapshot is the streamed raft_snapshot.dat (raft_snapshot.gob is migrated on start), and in raft mode the engine WAL also carries an applied-index marker so a restart never re-applies the log (details). Older builds cannot read the new files — see Raft log persistence.

With multiple disks (-data-dirs /mnt/nvme0,...,/mnt/nvme7), each disk gets its own independent WAL, VLog, segment file, and compaction goroutine. Shard i always lives on disk i % numDisks — a failure on one disk does not affect the other disks' shards.

Value transform pipeline

Write path:   plaintext  →  compress  →  encrypt  →  VLog bytes on disk
Read  path:   disk bytes →  decrypt   →  decompress →  plaintext

Compression runs before encryption: encrypted ciphertext has high entropy and is effectively incompressible, so compressing after encryption would waste CPU for no space savings. FlagCompressed and FlagEncrypted are stored per-record on the IndexEntry, so records written before encryption was enabled remain readable — the flags, not a global setting, decide which transforms the read path applies.

Compression uses zstd (level 1) by default, is attempted for values ≥ 256 bytes, and is kept only when it actually shrinks the value. Encryption uses AES-256-GCM with a unique 12-byte nonce per record, keyed from the VELTRIXDB_ENCRYPTION_KEY environment variable (base64) or a --encryption-key-path file.

Crash recovery

Every startup replays the WAL; after an unclean shutdown (crash, OOM kill, SIGKILL) that is the full write history since the last checkpoint:

1. replayWAL() opens wal.log on each disk.
2. Each record (binary or legacy text) is decoded and its CRC32C is checked. Crash replay, the PITR archiver and PITR restore share one decoder (walReader). A torn or corrupt record ends replay: everything before it is applied, and the server logs where it stopped.
3. applyWALReplay() rebuilds the in-memory sharded index in the background. The engine accepts traffic immediately; replay never overwrites a live write that arrived during warm-up.
4. For KV-separated records (vlogOffset > 0), the VLog already holds the value bytes — the WAL entry only re-establishes the index pointer, without re-reading the value.
5. Legacy 6-, 7-, 8- and 10-field text entries are still parsed for backward compatibility.
6. After replay the server rebuilds the search indexes from the persisted @vec/, @txt/ and @vecns/ keys and refuses searches until that finishes (INFO → search_ready); see Search subsystem.

On clean shutdown (SIGTERM) the checkpoint described above has already rewritten wal.log as one record per live key, so the next startup replays O(live keys) records rather than the full write history. The checkpoint uses the current --wal-format.

If a process crashes after the VLog write but before the WAL write completes, the value bytes are orphaned but were never referenced by the index — they are simply reclaimable garbage on next GC, and the client never received an acknowledgment. This ordering (VLog write, then WAL write) is what keeps crash recovery safe without requiring the two writes to be atomic with each other.