Key–value separation, WiscKey-style.

LSM trees rewrite values over and over as compaction merges levels — typically a 10–30× write amplification tax on NVMe endurance and tail latency. We split keys from values.

Each of the 8192 index shards holds a 64-byte record per key — disk offset, size, TTL, version. Values live in an append-only VLog on NVMe, one per disk, written once; there is no LSM compaction. Value-log GC runs in the background, copies only still-live values out of mostly-dead segments, and throttles itself when read latency rises — it is off the read path.

That makes write amplification close to 1× by design — the only rewrite is GC relocating live data. We have not published a write-amplification measurement; the performance page lists what we have measured and under which conditions.

An index the garbage collector never scans.

A Go map holding hundreds of millions of index entries is something the GC has to walk. Since v1.6 the index lives off the Go heap entirely.

Since v1.6: on cgo builds (Linux and macOS, and the Docker image) each shard's index is a C++ table — 64-byte records, 8-byte tag/position slots, a per-shard key arena, large arrays individually mmap'd. Measured with 5M keys: full GC 21 ms → 0.27 ms, settled RSS 168 → 142 B/key, for ~19 ns more per cache-miss lookup. VELTRIXDB_INDEX=map opts out; CGO_ENABLED=0 builds keep the Go map. Hosts need vm.max_map_count ≥ 262144 at large key counts.

The rest of the hot path is deliberately plain. Reads are a Go pread, 4 KB-aligned, checked against magic + CRC32C. VLog batches are one contiguous pwrite and one fdatasync; the WAL flusher does one write(2) per group-commit batch. An io_uring bridge for VLog batch writes exists but is opt-in and off by default (VELTRIXDB_URING_BRIDGE): with a batch already down to one pwrite, it saves at most one syscall. The full picture is in the technical whitepaper.

What shipped, and what didn't pay off.

v1.6 through v1.12 carry engine changes we measured — including one that did not pay off overall. v1.13 (2026-10-01) was documentation and test fixes over v1.12; v1.14 (2026-10-05) and v1.15 (2026-10-07) fix correctness gaps and make Raft persistence incremental; v1.16 (2026-10-08, the latest release) adds mutex / block profiling flags and striped read counters.

Binary WAL (default since v1.6). 48-byte header, CRC32C over the whole record. Keys containing | or a newline no longer cut replay short — with the old text format, an acknowledged write after such a key could be lost on a crash restart. Encoding is 4× faster per record, replay 2.5× faster; old text WALs still replay. Ordered index: overwrites of a live key no longer re-insert into the skiplist — server batch writes went from 1.45M to 3.54M keys/s (8 clients × 1024-key MPUT, 1M keys, macOS).

Adaptive group commit (default since v1.12). The WAL and VLog flush windows became upper bounds: a lone writer is synced at once, concurrent writers after an idle gap of about one fdatasync; --group-commit=fixed restores the full-window wait. With an emulated 300 µs fdatasync on a macOS laptop (15 ms window, 128 B values): one writer’s P50 went from 16 ms to 0.45 ms, and 64 writers from 4.0K to 37.5K durable writes/s at P99 2.6 ms. It has not yet been measured on Linux NVMe. Raft read barrier (v1.12): a newly elected leader now waits for its no-op before serving local reads, and MGET passes the same barrier as GET.

C++ network front-end (experimental, opt-in). --net=cpp|uring|poll runs one event loop per core for the binary PUT GET DEL PING MPUT MGET commands, standalone mode only; reads that need disk are handed to goroutines. On a 4-CPU CI runner (measured before disk reads moved off the loop) it lowered read P50 (154 vs 286 µs) but had a worse read P99 and ~12% lower batch-write throughput than the default Go listener. It does not serve the search commands. It is not faster overall, so --net=go stays the default.

HNSW in RAM, exact re-rank from NVMe.

Each namespace has one HNSW graph (cosine similarity; M = 16, efConstruction = 200, ef 64 by default, tunable per query with EF). The full float32 vectors always live in the value log; how much of them RAM holds is a per-namespace choice.

Memory layouts (VCREATE ns dim QUANT none|int8|pq GRAPH memory|disk), measured as Go heap per 768-dim vector over 10K vectors on macOS: float32 3,383 B; int8 1,078 B, top 4 × k candidates re-ranked from disk; PQ (m = 96) 493 B, the whole beam re-ranked; PQ + GRAPH disk 378 B of heap plus 216 B in a memory-mapped file the kernel can page to NVMe. The re-rank reads the candidates’ float32 vectors with one parallel MultiGet.

Text and hybrid: Okapi BM25 (k1 = 1.2, b = 0.75) over lowercase runs of Unicode letters and digits — Indic scripts stay whole; no stemming, stop words or phrase queries. HSEARCH fuses with RRF, score = α/(60 + rank_vec) + (1 − α)/(60 + rank_text). Clusters: searches fan out to every non-failed node and merge; BM25 sums every node’s statistics first. A peer that misses --search-timeout-ms (2000) fails the request and is named, unless --search-allow-partial. The node-to-node listener is signed with HMAC-SHA256 via --cluster-secret-file, or use mTLS; without either the server warns it is unauthenticated.

What it won’t do yet: the graph and inverted index are rebuilt at every start and searches are refused until that ends; tested to 100K real vectors and 10K at 768-dim, nothing at 10M+; one graph per namespace, cosine only, no GPU; on v1.13 and earlier DEL removes the record but not its vector or text — use VDEL / TDEL too (Fixed in v1.14: DEL deletes them in every namespace); the search API is in the bundled Go client only (other languages use the text protocol), and the --net=cpp front-ends do not serve it.