Architecture Overview

How a write and a read actually move through VeltrixDB, from the TCP listener down to NVMe.

System diagram

The serving path is Go, with two C++ exceptions on a cgo build: the per-shard index tables live off the Go heap in C++ (default on every cgo build with Go ≥ 1.21, macOS included), and --net=cpp swaps in an opt-in C++ network front-end. Dashed boxes are optional / opt-in.

Client (any SDK or nc)
        │ TCP                                  ┆ TCP (opt-in)
        ▼                                      ▼
┌──────────────────────────────────┐  ┌┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┐
│ TCP Server (cmd/server)          │  ┆ C++ network front-end (netfront/)┆
│ --net=go (default)               │  ┆ --net=cpp | uring | poll         ┆
│ 1 goroutine per conn             │  ┆ event loop per core              ┆
│ binary + text protocol           │  ┆ binary PUT GET DEL PING MPUT MGET┆
│ PUT/GET coalescing, cap 256      │  ┆ disk-bound reads → goroutines    ┆
│   → MultiPut / MultiGet          │  ┆ standalone only                  ┆
└────────────────┬─────────────────┘  └┄┄┄┄┄┄┄┄┄┄┄┄┄┄┬┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┘
                 │                                   ┆ vxnfExec, once per
                 ▼                                   ┆ loop iteration
┌────────────────────────────────────────────────────▼────────────┐
│ Storage Engine (storage/engine.go)                              │
│ Shard routing: FNV-1a(key) & 0x1FFF → 0..8191                   │
│                                                                 │
│ Index Vault              Bloom filter       Sharded LIRS cache  │
│ 8192 shards, RWMutex     lock-free,         up to 256 caches,   │
│ 64 B entry, no values    per shard          keyed on high bits  │
│ off-heap C++ table on cgo builds,                               │
│ Go map when CGO_ENABLED=0                                       │
│                                                                 │
│ Value transform: compress → encrypt (single chokepoint)         │
└────────────────────────────────┬────────────────────────────────┘
                                 ▼
┌─────────────────────────────────────────────────────────────────┐
│ Per NVMe disk — N independent sets                              │
│ WAL: binary records, CRC32C, one write + one fdatasync / batch  │
│ VLog: append-only, lock-free reservation,                       │
│       one pwrite + one fdatasync per batch                      │
│ Segment writer (O_DIRECT)                                       │
│ ── Defragmenter + VLog GC (3-tier) · Scrubber (CRC32C walk)     │
└─────────────────────────────────────────────────────────────────┘

┌┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┐
┆ C++ storage extras — Linux + CGO_ENABLED=1 (Docker image default)┆
┆ batch engine + NUMA pinning (BatchPut / WriteBatcher only)       ┆
┆ io_uring VLog write bridge — opt-in, VELTRIXDB_URING_BRIDGE      ┆
┆ VELTRIXDB_DISABLE_CGO_ENGINE=1 turns these off                   ┆
└┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┘

New in v1.6–v1.7 The native index default, the binary WAL, the opt-in io_uring bridge and the --net C++ front-end shipped in v1.6 (native index, binary WAL) and v1.7 (opt-in io_uring bridge, --net front-end).

Sharding

All data is split across 8192 shards. Each shard has its own lock, so a read on shard 5 never blocks a read on shard 42.

key → FNV-1a hash → bottom 13 bits (& 0x1FFF) → shard ID (0–8191)
shard ID % numDisks → which disk (WAL + VLog + compaction)

With 8 disks, 1024 shards land on each disk. A failure on one disk doesn't affect the other 7. The Go numShards constant and the C++ kNumShards constant must stay equal — they are a hard cross-language invariant.

Key components

Write path

What happens on PUT user:42 value:

1. Hash the key → shard 450 (FNV-1a & 0x1FFF) → disk 2 (450 % 8, with 8 disks)
2. Compress, then encrypt the value (transformForWrite, the single chokepoint)
3. Atomically reserve a VLog offset and write the value
4. Append a WAL record (key, VLog offset, on-disk length, transform flags, version)
5. WAL and VLog fdatasync race concurrently — the client waits for the slower of the two, not their sum
6. Install the shard 450 index entry (disk offset, disk index)
7. Insert the value into the LIRS cache
8. Return OK

The batch path (MultiPut) keeps the same order per disk, with one pwrite for the VLog extent and one WAL item for the batch. New in v1.6 It refreshes only keys already in the cache (PutIfPresent) instead of inserting, so bulk writes do not evict the read working set; a single Put stays write-through. The ordered (range-scan) index is touched only when a key becomes live, not on overwrites.

Read path

What happens on GET user:42:

1. Hash the key once → cache shard (high bits) and index shard 450 (low 13 bits)
2. LIRS cache hit → return the value with no disk I/O (~92 ns in-process)
3. Cache miss → Bloom filter says "definitely absent" → NOT_FOUND with no index lookup
4. Otherwise take a read lock on shard 450 and look up the index entry; absent → NOT_FOUND, expired TTL → expired
5. Found → read the value from the VLog at the recorded disk offset: a Go 4 KB-aligned pread that verifies magic and CRC32C
6. Decrypt → decompress, then insert the value into the cache
7. Return the value

New in v1.7 StorageEngine.GetNoIO runs the same flow but stops before the VLog read and returns needIO=true; GetAfterNoIO finishes the key. The opt-in C++ network front-end uses it so disk reads never run on an event-loop thread.

Admission control

Background garbage collection competes with foreground reads for NVMe bandwidth. VeltrixDB tracks an EWMA of read latency and throttles GC before reads degrade:

See Admission Control & GC for the full escalation ladder, including the emergency GC tier that bypasses admission pauses to prevent a garbage death-spiral.

C++ layer and network front-ends

The Go layer is fully functional on its own. C++ is optional, and less of it is wired up than the cpp/ file tree suggests:

The VLog read path is Go (storage/vlog.go ReadValue), not io_uring. VELTRIXDB_DISABLE_CGO_ENGINE=1 turns off the batch engine and bridge and selects the Go map index.

Network front-ends (--net) New in v1.7

--net=go (default) is one goroutine per connection and serves every protocol and command. --net=cpp / uring / poll selects the C++ front-end in netfront/: one event loop per core (--net-threads, default NumCPU), each with its own SO_REUSEPORT listener, io_uring on Linux and poll() elsewhere. Each loop iteration calls Go once (vxnfExec) for all parsed requests. Reads that need no disk are answered inline via GetNoIO; reads that need a VLog read, and all writes, run on goroutines, and a connection with such a request in flight is not parsed further, so each connection stays strictly ordered. At shutdown the server logs a per-stage latency report.

Search subsystem New in v1.11–v1.12

Vector, full-text and hybrid search are derived state over reserved keys. A vector, a text document and the KV record that share an id describe one thing; the vector and document are stored as ordinary keys, and the in-RAM indexes are built from them:

PUT / Delete / MultiPut of a reserved key
  @vec/<ns>/<id>  float32 blob → HNSW index of <ns>
  @txt/<ns>/<id>  UTF-8 text   → BM25 index of <ns>
  @vecns/<ns>     settings     → create / re-encode <ns>
  @idxdef/<name>  field        → create / drop a
                                 secondary index
       │
       └─ storage/search_hooks.go runs after the
          write commits

• Hooks, not call sites. Put, Delete and MultiPut call the search hooks after the write commits, so the indexes follow the KV state whichever path wrote it — the API, Raft apply, replication apply, snapshot restore, partition transfer on rebalance, or backup restore. A hook failure is logged; the KV write has already committed.
• HNSW per namespace (cosine; M = 16, 32 on layer 0, efConstruction = 200, efSearch = 64). Each namespace keeps float32 vectors in RAM by default, or int8 / product-quantized codes that re-rank candidates against the float32 vectors read back from the VLog; from v1.12 the layer-0 edges can live in a memory-mapped file (GRAPH disk). Deletes are tombstones compacted in the background.
• BM25 (k1 = 1.2, b = 0.75) over a Unicode tokenizer, and hybrid search that fuses the vector and text rankings by weighted reciprocal rank (rrfK = 60, α default 0.5).
• Rebuild on restart. The graphs and the inverted index are not persisted. After WAL replay, RebuildSearchIndexes applies the @vecns/ settings and then loads every vector and document on all cores. Until it finishes, searches are refused (search indexes are still rebuilding after restart) unless --search-allow-partial; INFO reports search_ready=0 search_loaded=x/y, then search_ready=1. KV traffic is not affected.
• Clusters. The ring routes @vec/, @txt/ and @idx/ keys as the record they belong to, so filters run locally on each node, and each search fans out to every node — see Cluster & Consensus.

Commands, memory layouts, filters and measured recall are in Vector & Hybrid Search.

Cluster layer at a glance

cluster/      partition_map.go      consistent-hash ring
                                    (FNV-1a + fmix64, 64 vnodes/node),
                                    256 partitions, RoutingKey
              epoch.go              epoch fencing (stale → HTTP 409)
              failure_detection.go  Active → Suspect (3 s)
                                    → Failed (10 s) → Recovering
                                    → Active (2 good pings; since v1.14)
              gossip.go             TCP gossip + digest exchange
              partition_transfer.go key migration (/transfer/keys)

replication/  eventual / quorum / strong replication
              anti-entropy (re-sends un-acked writes)
              FAILED-replica recovery (since v1.14)

consensus/    Raft — election, log replication, snapshots,
              ReadIndex, single-server membership changes

Full detail in Cluster & Consensus, including which deployment mode (standalone / raft / replicated) gives you which consistency guarantee.

Next steps