Cluster & Consensus

VeltrixDB distributes data across nodes with a consistent-hash partition map, detects failures via gossip, and offers three deployment modes with distinct consistency guarantees — from single-node linearizability to Raft-backed quorum writes to configurable async/quorum/strong replication.

Consistent-hash partition map

The ConsistentHashRing (cluster/partition_map.go) maps a 64-bit hash space onto physical nodes using virtual nodes:

• Keys are hashed with FNV-1a, and (since v1.1.0) the result is passed through the fmix64 finalizer (the MurmurHash3 64-bit finalization step) to a uint64.
• Each physical node is assigned 64 virtual nodes on the ring by default.
• Routing is a binary search for the first virtual-node hash ≥ the key's hash, wrapping around to the start of the ring if none is found.

The finalizer matters: raw FNV-1a has weak avalanche for inputs that differ only in a short suffix, so sequential key patterns (user:0001…user:0999) landed within a few multiples of the FNV prime of each other — far closer together than one vnode arc — and entire sequential runs collapsed onto one node. fmix64 makes every input bit affect every output bit, restoring uniform placement for sequential keys. Because both the server and the cluster client route through the same cluster.HashKey function, they stay mutually consistent — but changing this hash re-shards ring placement (the v1.1.0 switch was made as a pre-GA breaking change for exactly that reason).

Virtual nodes solve two problems that a naive one-hash-per-node ring has: they even out load across physical nodes (more virtual nodes per physical node means a more even share of the ring), and they make rebalancing gradual — adding one physical node moves roughly 1/N of the ring's data instead of a bulk reshuffle.

Above the ring, a PartitionMap divides the hash space into 256 fixed partitions, each with a primary node and up to ReplicationFactor - 1 replica nodes (default replication factor: 3). Every mutation to the partition map — node add, node remove, state change, rebalance — increments a monotonic Version.

Rack-aware replica placement (v1.1.0)

Placement is failure-domain-aware when nodes are started with --rack-id <zone> (e.g. --rack-id=asia-south1-a): partition placement during rebalance and GetReplicasForKey never put two copies of the same partition in the same rack — unless the replication factor exceeds the number of distinct racks, in which case placement falls back to filling with distinct nodes while still spanning every rack available. Rack labels propagate between members via gossip digests (a node's self-reported entry is authoritative; third-party entries only fill blanks), so each node only needs its own flag, and /admin/cluster reports a "rack" field per node. Clusters with no rack labels keep the exact previous deterministic round-robin placement.

Epoch fencing

Because the partition map version increases on every topology change, nodes can detect when they are routing against a stale view: if a node's local PartitionMap.Version is behind the cluster's current version, it fetches the latest map before continuing to route. This epoch mechanism prevents a node that missed a topology update from silently routing requests to the wrong owner during a rebalance.

Gossip-based failure detection

Cluster topology and node health propagate via epidemic gossip: every node sends its view of cluster state to 3 random peers every second. A receiver that sees a higher partition-map version than its own fetches the latest map. Convergence across the cluster happens in O(log N) gossip rounds — a 100-node cluster converges in roughly 7 rounds (~7 seconds at a 1-second gossip interval).

The heartbeat state machine

Each node cycles through a small set of states, driven by how long it's been since the last heartbeat was observed:

The two-stage SUSPECT → FAILED transition (3 s, then 10 s) exists to absorb transient network blips without immediately evicting a healthy node from routing — a brief GC pause or packet loss burst produces a SUSPECT flap, not a full failover. A background recovery worker pings FAILED nodes periodically (up to 3 attempts, 5 s apart); if a node comes back it transitions through RECOVERING back to ACTIVE, triggering a rebalance.

Deployment modes and consistency guarantees

The distributed layer is wired into cmd/server through a write coordinator (cmd/server/coordinator.go) that fronts the storage engine. Every mutating client operation — PUT, DELETE, MultiPut, CAS, INCR, DECR, SETNX, TXN, and (since v1.1.0) namespace, hash-field, vector, secondary-index, and list/set writes, on both text and binary protocols — goes through it. In raft mode each of these is a dedicated FSM op replayed deterministically on every member; in replicated mode they decompose to plain KV writes on composite internal keys (ns\x00key, key\x01field, @vec/…, key\x02seq, key\x03member) and ship as ordinary replication traffic. The --mode flag selects one of three behaviors:

standalone is the pre-existing single-node path with zero distributed-systems overhead. raft trades read freshness for strict write ordering and quorum durability. replicated trades strict write ordering for configurable, tunable ACK latency across copies.

Replication consistency levels (replicated mode)

The --consistency flag controls when the client receives its ACK:

• eventual (async) — ACK immediately after the local write; replicas catch up in the background.
• quorum — ACK only after a majority of replicas (counting the local copy) have applied the write.
• strong — ACK only after all replicas have applied the write.

quorum and strong surface explicit errors to the client when the required copies cannot be reached rather than silently downgrading the guarantee:

• ErrReplicationTimeout
• ErrQuorumNotReached

For example, a strong write attempted while a majority of replicas are down returns an error instead of ACKing with fewer copies durable than requested — the client's consistency contract is never silently weakened.

The write coordinator

The write coordinator is the single entry point every mutating operation passes through before reaching the storage engine, regardless of deployment mode. In standalone mode it is effectively a pass-through. In raft mode it submits the operation to the Raft log and blocks until the FSM has applied it. In replicated mode it performs the local write first, then hands the operation to the replication engine and blocks according to the configured consistency level.

Raft FSM: Apply, Snapshot, Restore

raftFSM (cmd/server/raft_fsm.go) implements consensus.SnapshotStateMachine:

• Apply decodes a gob-encoded write command and runs the engine's own Put/Delete/atomic/txn method directly. This is deterministic because Raft guarantees every node sees the identical log order — replaying the same sequence of engine calls on every node produces identical state.
• Results for operations that return a value or status (CAS, INCR, SETNX, TXN) are delivered back to the submitting client through a per-request result side-channel keyed by a node-unique request ID — the FSM's Apply itself has no direct return path to the original client connection.
• Snapshot / Restore dump and reload the entire keyspace using the engine's paginated ScanCursor, allowing log compaction without keeping the full history of every write.

Non-leader nodes reject writes with a MOVED <leader-addr> redirect rather than silently forwarding or queuing them — this keeps the write path simple and pushes leader discovery to the client (see the cluster-aware client below).

Linearizable reads: the ReadIndex fence

A local read on the Raft leader can still be stale: a partitioned ex-leader may keep serving reads after a new leader has committed writes elsewhere. --linearizable-reads (v1.1.0, raft mode only) closes that hole with the Raft §6.4 ReadIndex protocol (consensus/read_index.go), which never writes to the log:

1. Capture the fence. The leader records its current commitIndex as the read fence. The fence is only valid once the leader has committed at least one entry in its own term (otherwise commitIndex may predate a newer leader's writes) — immediately after an election the read returns a retryable ErrLeadershipUnconfirmed until the election no-op commits.
2. Confirm leadership with a quorum heartbeat round. The leader sends a fresh AppendEntries heartbeat to every peer; any reply whose term does not exceed the leader's counts as an acknowledgement (a log-lagging follower still recognises the leader). Reaching a quorum proves no higher-term leader existed at the moment of the round. A reply with a higher term deposes the leader instead.
3. Wait for apply catch-up, then read. The read blocks until the local state machine has applied up to the fence, then the coordinator performs the actual storage read.

Cost and routing: one heartbeat round-trip per GET (a single-node cluster skips the round — the leader is the quorum), plus any apply catch-up, bounded by a 2-second per-read timeout (ErrReadIndexTimeout). Followers reject linearizable reads with a MOVED <leader-addr> redirect, exactly like writes.

Auto-rebalance and the TransferAgent

Before v1.1.0, cluster/partition_transfer.go was fully implemented but never instantiated by the server: a node join or leave updated the ring (routing) while the data stayed put. cmd/server/rebalancer.go closes that gap by subscribing to PartitionMap membership events (SubscribeMembership) and wiring them to physical key migration:

PartitionMap membership event (add / remove / state change)
     │  debounce 3 s — coalesce a burst of gossip events
     ▼
pm.Rebalance(pm.PartitionCount())   — recompute partition ownership
ta.MigrateToNewOwners()             — stream re-owned keys, delete local copies

• Event filtering: local heartbeat state flaps don't move data; only topology-relevant transitions (joins, removals, hard failures, recoveries) reset the debounce window.
• Transfer transport: the TransferAgent runs a dedicated partition-transfer HTTP listener on client port +5 (override with -transfer-addr); when the -cluster-tls-* flags are set, transfers use the same inter-node TLS/mTLS configuration.
• Safe retry: MigrateToNewOwners deletes a local key only after the destination confirms receipt, so a failed migration is retried on the next membership event and re-sends only the unconfirmed remainder.
• Opt-out: --auto-rebalance=false disables the automatic trigger (the transfer listener still starts, so other nodes can push keys to this one).

The cluster-aware client

client.Client (client/client.go) is a real topology-aware client, not a dumb connection wrapper:

• It fetches cluster topology over the storage port via the TOPOLOGY command (also exposed at /admin/cluster).
• It builds a consistent-hash ring identical to the server's — same cluster.HashKey function, same 64 virtual nodes per physical node — so it can compute key ownership locally instead of asking the server on every request.
• It routes each key directly to its owning node using that locally-built ring.
• It follows MOVED redirects to find the current Raft leader when a write lands on a non-leader node.

/admin/cluster additionally reports node role, Raft term and leader, peer list, partition epoch, and per-replica replication lag — useful both for the client's own topology refresh and for operator diagnostics.

Remaining gaps

Everything else that was node-local in the previous phase — namespace, hash-field, vector, secondary-index data, and list/set writes, plus linearizable reads and physical data rebalancing — is now wired end-to-end through the coordinator, ReadIndex, and TransferAgent paths described above.