Cluster & Consensus

VeltrixDB distributes data across nodes with a consistent-hash ring, detects failures via gossip, fans searches out to every node, and offers three deployment modes with distinct consistency guarantees — from single-node linearizability to Raft-backed quorum writes to configurable async/quorum/strong replication.

Consistent-hash partition map

The ConsistentHashRing (cluster/partition_map.go) maps a 64-bit hash space onto physical nodes using virtual nodes:

• Keys are hashed with FNV-1a, and (since v1.1.0) the result is passed through the fmix64 finalizer (the MurmurHash3 64-bit finalization step) to a uint64. What is hashed is RoutingKey(key): the key itself, except that derived search keys route as their record — @vec/<ns>/<id> and @txt/<ns>/<id> as <id>, @idx/<rule>/<value>/<primary> as <primary> — so a record, its vector, its document and its index entries stay on one node.
• Each physical node is assigned 64 virtual nodes on the ring by default.
• Routing is a binary search for the first virtual-node hash ≥ the key's hash, wrapping around to the start of the ring if none is found.

The finalizer matters: raw FNV-1a has weak avalanche for inputs that differ only in a short suffix, so sequential key patterns (user:0001…user:0999) landed within a few multiples of the FNV prime of each other — far closer together than one vnode arc — and entire sequential runs collapsed onto one node. fmix64 makes every input bit affect every output bit, restoring uniform placement for sequential keys. Because both the server and the cluster client route through the same cluster.HashKey function, they stay mutually consistent — but changing this hash re-shards ring placement (the v1.1.0 switch was made as a pre-GA breaking change for exactly that reason).

Virtual nodes solve two problems that a naive one-hash-per-node ring has: they even out load across physical nodes (more virtual nodes per physical node means a more even share of the ring), and they make rebalancing gradual — adding one physical node moves roughly 1/N of the ring's data instead of a bulk reshuffle.

Above the ring, a PartitionMap divides the hash space into 256 fixed partitions, each with a primary node and up to ReplicationFactor - 1 replica nodes (default replication factor: 3). Every mutation to the partition map — node add, node remove, state change, rebalance — increments a monotonic Version, reported in topology and gossip but not used to reject anything. The partition table is metadata (topology views, admin); key placement and migration use the ring.

Membership and client-side routing

On v1.16 and earlier membership is static: each server registers the nodes named in --peers id@host:port,... (or --seeds) at startup. There is no command or admin endpoint that adds, drains or removes a node at runtime — to add a node, restart every node with it in --peers. New on main after v1.16 — ships in the next release nodes are added and removed online through /admin/cluster/members and veltrix cluster, and the persisted membership wins over --peers once a change was made (see Online membership changes). Gossip still does not carry membership. The server also does not forward a request to the key's owner: routing is done by the cluster-aware client (below). In raft mode the leader takes every write and followers answer MOVED.

Rack-aware replica placement (v1.1.0)

Placement is failure-domain-aware when nodes are started with --rack-id <zone> (e.g. --rack-id=asia-south1-a): partition placement during rebalance and GetReplicasForKey never put two copies of the same partition in the same rack — unless the replication factor exceeds the number of distinct racks, in which case placement falls back to filling with distinct nodes while still spanning every rack available. Rack labels propagate between members via gossip digests (a node's self-reported entry is authoritative; third-party entries only fill blanks), so each node only needs its own flag, and /admin/cluster reports a "rack" field per node. Clusters with no rack labels keep the exact previous deterministic round-robin placement.

Epoch fencing

The fencing check is a separate membership epoch (cluster/epoch.go), advanced by every AddNode / RemoveNode and by newer epochs seen in gossip. Every key-migration batch carries the sender's epoch; a receiver whose epoch is newer refuses it with HTTP 409, so a node with a stale view cannot push keys into a cluster that has moved on. No node fetches a partition map from a peer.

Gossip-based failure detection

Node liveness propagates via epidemic gossip: every second a node advances its own heartbeat counter and exchanges a JSON digest with 3 random peers over one short TCP connection each. A node whose heartbeat counter advanced in a digest counts as alive, so liveness spreads transitively. A receiver adopts gossip addresses and racks (self-reported entries win) and newer epochs, ignores nodes it does not know, and does not adopt the sender's view of node states — each node derives states locally from heartbeat ages, checked every second.

The heartbeat state machine

Each node cycles through a small set of states, driven by how long it's been since the last heartbeat was observed:

The two-stage SUSPECT → FAILED transition (3 s, then 10 s) exists to absorb transient network blips without immediately evicting a healthy node from routing — a brief GC pause or packet loss burst produces a SUSPECT flap, not a full failover. A background recovery worker pings a FAILED node (TCP PING, 2 s timeout) up to 3 times back to back; if it answers, the node becomes RECOVERING and a rebalance is triggered. If all attempts fail it stays FAILED until a heartbeat arrives.

Deployment modes and consistency guarantees

The distributed layer is wired into cmd/server through a write coordinator (cmd/server/coordinator.go) that fronts the storage engine. Every mutating client operation — PUT, DELETE, MultiPut, CAS, INCR, DECR, SETNX, TXN, and (since v1.1.0) namespace, hash-field, vector, secondary-index, and list/set writes, on both text and binary protocols — goes through it. In raft mode each of these is a dedicated FSM op replayed deterministically on every member; in replicated mode they decompose to plain KV writes on composite internal keys (ns\x00key, key\x01field, @vec/…, key\x02seq, key\x03member) and ship as ordinary replication traffic. The --mode flag selects one of three behaviors. Raft and replicated mode are mutually exclusive: the server builds either a Raft coordinator or a replicated one, never both, and in raft mode no replication engine exists.

standalone is the pre-existing single-node path with zero distributed-systems overhead. raft trades read freshness for strict write ordering and quorum durability. replicated trades strict write ordering for configurable, tunable ACK latency across copies.

New in v1.7 The opt-in C++ network front-end (--net=cpp|uring|poll) writes straight to the engine, bypassing Raft/replication routing, so the server refuses it with --mode=raft or --mode=replicated; cluster modes always run on the default --net=go.

Replication consistency levels (replicated mode)

The --consistency flag controls when the client receives its ACK:

• eventual (async, the server default) — ACK immediately after the local write; replicas catch up in the background.
• quorum — ACK only after a majority of replicas (counting the local copy) have applied the write.
• strong — ACK only after all replicas have applied the write.

quorum and strong surface explicit errors to the client when the required copies cannot be reached rather than silently downgrading the guarantee:

• ErrReplicationTimeout
• ErrQuorumNotReached

For example, a strong write attempted while a majority of replicas are down returns an error instead of ACKing with fewer copies durable than requested — the client's consistency contract is never silently weakened.

The write coordinator

The write coordinator is the single entry point every mutating operation passes through before reaching the storage engine, regardless of deployment mode. In standalone mode it is effectively a pass-through. In raft mode it submits the operation to the Raft log and blocks until the FSM has applied it. In replicated mode it performs the local write first, then hands the operation to the replication engine and blocks according to the configured consistency level.

Raft FSM: Apply, Snapshot, Restore

raftFSM (cmd/server/raft_fsm.go) implements consensus.SnapshotStateMachine:

• Apply decodes a gob-encoded write command and runs the engine's own Put/Delete/atomic/txn method directly. Raft guarantees every node sees the identical log order, so replaying the same sequence of engine calls produces identical state — as long as no Apply decision reads node-local state. On v1.16 and earlier some do: each node stamps versions and TTL deadlines with its own clock (see Deterministic Apply).
• Results for operations that return a value or status (CAS, INCR, SETNX, TXN) are delivered back to the submitting client through a per-request result side-channel keyed by a node-unique request ID — the FSM's Apply itself has no direct return path to the original client connection.
• Snapshot / Restore dump and reload the entire keyspace, allowing log compaction (default threshold 8192 entries) without keeping the full history of every write. On v1.16 and earlier: the snapshot is read with the engine's paginated ScanCursor while Apply waits, it is capped at 256 MB (a larger one is skipped and the log keeps growing), TTLs are not preserved — restored keys become immortal — and a restore only writes the snapshot's pairs, so keys deleted while a follower lagged survive on it. Fixed on main after v1.16 — ships in the next release see Streaming snapshots.
• ApplyBatch (raft_fsm_batch.go) coalesces consecutive committed PUTs into one MultiPut; other ops break the run and apply singly, in log order.

Non-leader nodes reject writes with a MOVED <leader-addr> <leader-id> redirect rather than silently forwarding or queuing them — this keeps the write path simple and pushes leader discovery to the client (see the cluster-aware client below).

Raft log persistence and follower durability

Each node keeps its Raft state in <--data>/raft/. On v1.14 and earlier the log, term and vote are one file, raft_state.gob, and every flush gob-encodes and fsyncs the whole retained log into it (up to 8192 entries, about 8 MB with ~1 KB values), on followers even for heartbeats. A follower also writes and fsyncs each AppendEntries while holding the node lock, so heartbeats, votes and follower reads wait behind that fsync.

Fixed in v1.15 (consensus/raft_storage.go, consensus/pipeline.go):

• Incremental log. raft_log.dat is append-only with a CRC32C per record: a commit writes only its new records and one fsync, whatever the length of the retained log. Term and vote are in raft_meta.dat, replaced atomically (temp file, fsync, rename) only when they change and before the node acts on them. A torn tail is truncated on load, and compaction rewrites the log atomically after the snapshot is durable. raft_snapshot.gob is unchanged in v1.15 and v1.16 (Fixed on main after v1.16 — ships in the next release replaced by raft_snapshot.dat, below).
• Followers acknowledge only durable entries. One syncer goroutine per node writes and fsyncs the queued records outside the node lock, sharing one fsync across every AppendEntries that arrived meanwhile. Heartbeats and votes are answered without waiting for a follower fsync. The leader counts itself toward a commit quorum only for entries its own fsync covered.
• Follower commit index. A follower takes the leader's commit index only up to the last entry the request verified (Raft Figure 2). Earlier builds capped it at the follower's whole log, which could commit a stale suffix from an older term.
• No MOVED loop after an election. A node clears its leader ID when its term advances, so a deposed leader no longer answers MOVED pointing at itself.

Measured on one laptop only (3 local processes sharing one drive, go-ycsb, 20K records, 16 threads, one run each; not a cluster result): raft load went from 70 to 1,046 inserts/s and workload A from 87 to 1,212 ops/s.

Pipelined replication: --raft-pipeline

Fixed in v1.15 --raft-pipeline (default false; env VELTRIX_RAFT_PIPELINE) controls how the leader sends AppendEntries. Off: one in flight per follower, sent once per leader fsync. On: up to 8 in flight per follower, sent before the leader's own fsync. Durability and commit rules are the same either way. It is off by default because, with three nodes sharing one disk on one laptop, pipelining was slower for raft load (560 vs 1,038 inserts/s, medians of 2 runs), though faster on workload B (10,547 vs 5,630 ops/s). It has not been measured on separate hosts: turn it on only after a multi-host benchmark on your hardware shows it ahead (see Benchmarking). It has no effect outside --mode=raft.

Streaming snapshots and chunked InstallSnapshot Fixed on main after v1.16 — ships in the next release

On v1.16 and earlier the snapshot is one in-memory blob in raft_snapshot.gob: a snapshot over 256 MB is skipped (snapshot too large ... skipping compaction) and the log then grows in memory and on disk without bound; the capture holds Apply for the whole keyspace scan; TTLs are dropped; a follower's restore only writes the snapshot's pairs, so every key deleted on the leader while it lagged (with its vectors, documents and @idx/ entries) survives on it; and secondary-index definitions are not carried. On main (consensus/snapshot.go, cmd/server/raft_snapshot.go):

• No size cap. Once the retained log reaches --raft-snapshot-threshold entries (default 8192, env VELTRIX_RAFT_SNAPSHOT_THRESHOLD, 0 = never) the state machine is streamed record by record to raft_snapshot.dat (checksummed header, records and trailer; temp file, fsync, rename, directory fsync). An existing raft_snapshot.gob is migrated on start.
• Capture does not stall writes. It runs beside Apply and holds the apply lock only to re-read the keys written in its last moments. The snapshot carries every key with its absolute TTL expiry, plus the secondary-index definitions.
• Chunked InstallSnapshot. A lagging follower receives the file in chunks (--raft-snapshot-chunk-kb, default 2048, env VELTRIX_RAFT_SNAPSHOT_CHUNK_KB), one in flight; lost, duplicated and reordered chunks, a leader change and a newer snapshot replacing the file mid-transfer are all handled. The follower restores without holding the raft lock, keeps answering heartbeats and does not start elections while it installs.
• Restore = snapshot. Keys absent from the snapshot are deleted, TTLs are restored exactly, keys that expired in between are skipped, and the search indexes are rebuilt from the restored keys (searches are refused meanwhile). An interrupted install is redone at the next start.

Laptop measurement, 1,000,000 keys: slowest write during a capture 2.76 s → 28 ms (Apply fence 0.2–0.3 ms); a follower catching up through InstallSnapshot 42 s → 2.6 s. Laptop numbers, not a cluster result.

Exactly-once apply across restarts Fixed on main after v1.16 — ships in the next release

Raft persists the log, term and vote, but on v1.16 and earlier not which entries the state machine applied. Every restart therefore re-applies the retained log (or the tail after the snapshot) over an engine that already holds it: INCR / DECR counters double (INCR ×10 then a clean restart reads 20), list pushes repeat, version-guarded TXNs fail their guard and relative TTLs restart; in a rolling restart each node ends with different counters.

On main the engine appends an applied-index marker to its WAL after each applied run (no extra fsync), and in --mode=raft WAL replay stops at the newest marker, so the engine holds exactly the entries up to it and raft skips them; a crash mid-entry rolls the entry back as a whole and it is applied again once. A node with a snapshot also skips the O(keys) startup restore when its engine is already past the snapshot (single node on an Apple-silicon laptop, snapshot of 1,000,000 × 128 B keys, two runs: raft start 1.62 s / 1.91 s → 78 ms / 145 ms; engine WAL replay unchanged). Upgrading a node with no marker: with a snapshot, the snapshot is restored and the tail applied; without one, the engine is reset and the whole log applied once. An older build ignores the marker; v1.16 shows it as one extra live key ("\x00veltrix/raft-applied"), and re-upgrading such a node needs no wipe.

Deterministic Apply Fixed on main after v1.16 — ships in the next release

On v1.16 and earlier the version token that VER / GETVER return and a TXN SETIF compares is each node's own write timestamp, so a guarded TXN applied through the log can hold on the node whose token the client read and fail on the others — the replicas diverge. Relative TTLs also become different deadlines on each node. On main the leader stamps every command with its clock and each node applies it at the derived raft time max(Ts, previous + 1): versions are the same on every node, monotonic per key, and survive restarts, snapshot installs and rebuilds; TTL deadlines and every expiry decision in Apply (SETNX, CAS, INCR, SETIF expecting 0, list and hash ops) use that time. Apply also waits for the engine's WAL replay, and namespace quotas are checked by the leader before it submits. Wire format unchanged.

Online membership changes New on main after v1.16 — ships in the next release

GET / POST /admin/cluster/members and DELETE /admin/cluster/members/{id} (admin auth), or veltrix cluster members | add-node id@host:port | remove-node id [--force]. Any node forwards a change to the raft leader or the replicated-mode coordinator over the transfer listener, signed with the cluster secret. Changes serialise: a second one while the first is uncommitted gets 409. Step-by-step procedures are in the membership runbook.

• Raft: learners. A new node is added as a learner — replicated to through entries or chunked InstallSnapshot, never counted toward a quorum — and promoted to voter once a catch-up round ends within 1 s. A change whose new voter set has no quorum the leader recently heard from is refused up front (503). A removed leader steps down once its removal commits. Raft mode still moves no keys.
• Raft: pre-vote. An election starts with a pre-vote round that members refuse for a non-member, a stale log, or while they hear from a leader. A removed, partitioned or not-yet-promoted node no longer bumps its term, so it cannot depose a working leader when it returns (on v1.16 and earlier any partitioned follower forces an election on rejoin).
• --join starts a node empty with an empty configuration: it never campaigns and votes for nobody until a leader adds it. --advertise-addr sets the client address other nodes dial. Both are ignored on a node named in --peers or with persisted membership.
• Replicated mode keeps a versioned membership document on every node (<--data>/cluster_membership.json); changes run on one coordinator (the lowest member ID), are pushed to every member and pulled by members that missed them. A new member gets each member's own writes through replication catch-up and every key through the RF-aware migration (put-if-absent). A node is drained before it is removed: it refuses writes until every write it originated is acknowledged by every replica, then sends its keys to the remaining members. A removed node refuses writes, also after a restart.
• Persisted membership wins over --peers in both modes once a change was made online.
• --force-new-cluster (dangerous, disaster recovery only): raft — reset this node's membership to itself after a quorum was lost for good; replicated — move the persisted membership aside and start from --peers.

Reads on a new leader: the no-op barrier New in v1.12

A newly elected leader appends a no-op entry in its own term (Raft §5.4.2). Until that no-op is applied, its state machine may not yet contain entries its predecessor committed and acknowledged, so a local GET right after failover could return "not found" for an acknowledged write. RaftNode.WaitLeaderApplied now holds reads on a leader until lastApplied reaches the no-op (bounded by 2 s); after that it is one atomic load. Followers never wait — their local reads are stale by design.

Every read that passes coordinator.readBarrier() gets this — on v1.13 and earlier, GET and every MGET path. In a unit test, a read right after failover missed the acknowledged write in 20 of 20 failovers without the barrier and 0 of 20 with it.

Linearizable reads: the ReadIndex fence

A local read on the Raft leader can still be stale: a partitioned ex-leader may keep serving reads after a new leader has committed writes elsewhere. --linearizable-reads (v1.1.0, raft mode only) closes that hole with the Raft §6.4 ReadIndex protocol (consensus/read_index.go), which never writes to the log:

1. Capture the fence. The leader records its current commitIndex as the read fence. The fence is only valid once the leader has committed at least one entry in its own term (otherwise commitIndex may predate a newer leader's writes) — immediately after an election the read returns a retryable ErrLeadershipUnconfirmed until the election no-op commits.
2. Confirm leadership with a quorum heartbeat round. The leader sends a fresh AppendEntries heartbeat to every peer; any reply whose term does not exceed the leader's counts as an acknowledgement (a log-lagging follower still recognises the leader). Reaching a quorum proves no higher-term leader existed at the moment of the round. A reply with a higher term deposes the leader instead.
3. Wait for apply catch-up, then read. The read blocks until the local state machine has applied up to the fence, then the coordinator performs the actual storage read.

Cost and routing: one heartbeat round-trip per GET or MGET (per read of any kind since v1.14) (a single-node cluster skips the round — the leader is the quorum), plus any apply catch-up, bounded by a 2-second per-read timeout (ErrReadIndexTimeout). Followers reject linearizable reads with a MOVED <leader-addr> <leader-id> redirect, exactly like writes.

Auto-rebalance and the TransferAgent

Before v1.1.0, cluster/partition_transfer.go was fully implemented but never instantiated by the server: a node join or leave updated the ring (routing) while the data stayed put. cmd/server/rebalancer.go closes that gap by subscribing to PartitionMap membership events (SubscribeMembership) and wiring them to physical key migration:

PartitionMap membership event (add / remove / state change)
     │  debounce 3 s — coalesce a burst of gossip events
     ▼
pm.Rebalance(pm.PartitionCount())   — recompute partition ownership
ta.MigrateToNewOwners()             — stream re-owned keys, delete local copies

• Event filtering: SUSPECT flaps don't move data. On v1.14 and earlier joins, removals and transitions to FAILED, ACTIVE or RECOVERING reset the debounce window. Fixed in v1.15 only joins and removals do; state changes never start a migration (the failure detector still recomputes the partition table, but no keys move).
• Transfer transport: the TransferAgent runs a dedicated partition-transfer HTTP listener on client port +5 (override with -transfer-addr); when the -cluster-tls-* flags are set, transfers use the same inter-node TLS/mTLS configuration.
• Safe retry: keys move in 500-key batches (POST /transfer/keys); MigrateToNewOwners deletes a local key only after the destination acknowledged its batch with HTTP 200 (Fixed in v1.15 only after every current replica of the key acknowledged it), so a failed migration is retried on the next membership event and re-sends only the unconfirmed remainder (Fixed on main after v1.16 — ships in the next release also retried with backoff, e.g. while the destination is still starting).
• Epoch fencing: each batch carries the sender's membership epoch; a receiver with a newer epoch refuses it with HTTP 409.
• Pinned keys: @vecns/<ns> (vector namespace settings) and @idxdef/<name> (secondary-index definitions) are needed on every node, so they are copied to each destination ahead of the migrated keys and never deleted locally.
• Search follows the keys: the destination's Put and the source's Delete run the search hooks, so migrated vectors and documents become searchable on the new owner and disappear from the old one.
• Authentication: with --cluster-secret-file (or VELTRIXDB_CLUSTER_SECRET; at least 16 bytes, whitespace trimmed, same on every node) every request on the transfer listener carries X-Veltrix-Cluster-Time and an HMAC-SHA256 X-Veltrix-Cluster-Auth over time, method, path and body, and is refused with HTTP 401 if it does not verify or is more than 5 minutes off. /transfer/health stays open. Without a secret or --cluster-mtls the server logs [transfer] WARNING: listener <addr> is unauthenticated ….
• Opt-out: --auto-rebalance=false disables the automatic trigger (the transfer listener still starts, so other nodes can push keys to this one). Fixed in v1.15 in --mode=raft the flag is a logged no-op and the listener refuses /transfer/keys.

Distributed search fan-out New in v1.11

Search writes (VSET, TSET, VCREATE, IDXCREATE, …) go through the normal raft / replicated write path. Search reads — every vector, text and hybrid search, plus QUERY and IDXQUERY — run on every non-failed node and are merged (cmd/server/search_fanout.go):

• Transport: peers are asked over the transfer listener (client port +5) at POST /internal/search, with the same TLS and HMAC signing as key migration.
• BM25 in two phases: nodes first sum their corpus statistics, so every node scores with the same cluster-wide idf.
• Fail closed: a peer that does not answer within --search-timeout-ms (2000, per phase) fails the whole request with an error naming it — search incomplete: n of m peers did not answer: … (retry, or start the server with --search-allow-partial). A peer still rebuilding its indexes after a restart fails it too. --search-allow-partial answers with the rest instead (logged).
• Nodes the failure detector marked FAILED are not asked, so after a node dies searches fail until it is marked failed (about 10 s), then run on the remaining nodes.
• --search-fanout=false keeps searches local — correct only if every node holds every record.

A search costs one round trip to every node, so its cost grows with the cluster. See Vector & Hybrid Search for commands and measurements.

The cluster-aware client

client.Client (client/client.go, the Go client in the server repo) is a real topology-aware client, not a dumb connection wrapper:

• It fetches cluster topology over the storage port via the TOPOLOGY command (also exposed at /admin/cluster).
• It builds a consistent-hash ring identical to the server's — same cluster.HashKey function (FNV-1a + fmix64), same 64 virtual nodes per physical node — so it can compute key ownership locally instead of asking the server on every request.
• It routes each key directly to its owning node using that locally-built ring.
• It follows MOVED redirects (up to 3 hops per request) to find the current Raft leader when a write lands on a non-leader node.

/admin/cluster additionally reports node role, Raft term and leader, peer list, partition epoch, and per-replica replication lag — useful both for the client's own topology refresh and for operator diagnostics.

Remaining gaps

Namespace, hash-field, vector, text, secondary-index and list/set writes, linearizable reads and physical data rebalancing are wired end-to-end through the coordinator, ReadIndex and TransferAgent paths described above.