Architecture Overview
How a write and a read actually move through VeltrixDB, from the TCP listener down to NVMe.
System diagram
Client (any SDK or nc)
│ TCP — binary or text protocol
▼
┌─────────────────────────────────────────┐
│ TCP Server (cmd/server) │
│ 1 goroutine/conn · pipeline coalescing │
└──────────────────┬──────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Storage Engine (storage/engine.go) │
│ │
│ In-Memory Index LIRS Cache │
│ 8192 shards hot values in RAM │
│ │
│ WAL (per disk) VLog (per disk) │
│ group-commit append-only values │
└──────────────────┬──────────────────────┘
│ (Linux production only)
▼
┌─────────────────────────────────────────┐
│ C++ Layer │
│ ART index · io_uring · NUMA pinning │
└─────────────────────────────────────────┘Sharding
All data is split across 8192 shards. Each shard has its own lock, so a read on shard 5 never blocks a read on shard 42.
key → FNV-1a hash → bottom 13 bits (& 0x1FFF) → shard ID (0–8191)
shard ID % numDisks → which disk (WAL + VLog + compaction)With 8 disks, 1024 shards land on each disk. A failure on one disk doesn't affect the other 7. The Go numShards constant and the C++ kNumShards constant must stay equal — they are a hard cross-language invariant.
Key components
| Component | Role |
|---|---|
| In-memory index | A hash map per shard (Go) or an Adaptive Radix Tree (C++). Each entry stores disk offset, disk index, value size, flags, and a monotone version counter — about 64 bytes per key. |
| LIRS cache | Scan-resistant eviction. Small values (≤256 B) get eviction priority 2 vs. priority 1 for larger values, so a big sequential scan can't push your hot working set out of RAM. Sized with -cache <MB>. |
| WAL (Write-Ahead Log) | Group-commit: many concurrent writers share one fdatasync call per flush window (default 10 ms). One WAL file per disk. |
| VLog (Value Log) | Append-only file per disk. Values live here; the index holds only metadata (this is the WiscKey KV-separation). Appends are lock-free via atomic offset reservation. |
| Block packing | Batched writes (MultiPut) pack up to 26 records per 4 KB VLog block. For 128 B values that's 152 bytes of overhead per record instead of 4096 — a 27× density improvement. A single Put stays on the unpacked fast path. |
Write path
What happens on PUT user:42 value:
- Hash the key → shard 307 → disk 2 (
307 % 8) - Atomically reserve a VLog offset and write the value
- Write a WAL entry (key + VLog offset)
- WAL and VLog
fdatasyncrace concurrently — the client waits for the slower of the two - Update the shard 307 index entry (disk offset, version++)
- Insert the value into the LIRS cache
- Return
OK
Read path
What happens on GET user:42:
- Hash the key → shard 307
- LIRS cache hit → return the value with no disk I/O (~220 ns)
- Cache miss → take a read lock on shard 307, look up the index entry
- Not in the index →
NOT_FOUND - In the index → read the value from the VLog at the recorded disk offset (~400 µs on NVMe)
- Insert the value into the cache
- Return the value
Admission control
Background garbage collection competes with foreground reads for NVMe bandwidth. VeltrixDB tracks an EWMA of read latency and throttles GC before reads degrade:
| Read EWMA | Action |
|---|---|
| < 3 ms | Normal — GC runs at full speed |
| ≥ 3 ms | GC bandwidth capped at 60 MB/s |
| ≥ 4 ms | GC paused entirely; each PUT sleeps 2 ms to shed load |
| < 2 ms (recovery) | Everything resumes at full speed |
| No reads for 4 minutes | EWMA is treated as stale and GC resumes regardless |
See Admission Control & GC for the full escalation ladder, including the emergency GC tier that bypasses admission pauses to prevent a garbage death-spiral.
C++ layer (Linux only)
On Linux, VeltrixDB loads an optional C++ acceleration layer for production throughput. The Go layer is fully functional on its own and is what runs on macOS and in development.
| Component | Replaces | Why it's faster |
|---|---|---|
| ART index | Go hash map | Faster range scans, prefix compression |
| io_uring scheduler | Blocking pwrite | 3-tier I/O priority — foreground reads always beat background compaction |
| SQPOLL VLog reader | Blocking pread | Zero-syscall NVMe reads |
| NUMA thread pinning | Default OS scheduler | Eliminates cross-socket memory latency |
The C++ layer shaves roughly 25 µs off the NVMe read P99 on production hardware.
Cluster layer at a glance
cluster/ partition_map.go consistent-hash ring (FNV-1a, 64 vnodes/node), epoch fencing
failure_detection.go heartbeat SUSPECT → FAILED state machine
gossip.go TCP gossip listener + digest exchange
replication/ async / quorum / strong replication modes
vector clocks, anti-entropy, tombstone watermarks
consensus/ Raft — leader election + log replication + snapshotsFull detail in Cluster & Consensus, including which deployment mode (standalone / raft / replicated) gives you which consistency guarantee.