Deployment Modes
VeltrixDB serves every request from local state by default. The distributed layer is opt-in via the -mode flag and is wired directly into the serving path: every mutating request — plain KV (PUT/DELETE/MultiPut), atomics, TXN, and (since v1.1.0) namespace, hash-field, vector, secondary-index, and list/set writes — passes through a write coordinator in front of the storage engine, on both the text and binary protocols.
The three modes
Set the mode with -mode standalone|raft|replicated. Here is exactly what each mode guarantees for writes and reads — this is the load-bearing reference for choosing a mode, so read it precisely rather than skimming.
-mode | How writes are handled | Consistency guarantee |
|---|---|---|
standalone default |
Straight to the local engine — byte-for-byte the pre-existing single-node path. No Raft, no replication, no redirects. | Single-node linearizable (one writer, one copy). |
raft |
Ops are gob-encoded and submitted to a Raft log, committed by quorum, and applied on every node via a storage-backed finite state machine. Non-leaders reject writes with a MOVED <leader-addr> redirect. |
Linearizable writes (single Raft group, quorum commit). Reads default to local applied state — fast but possibly stale. With --linearizable-reads, GET runs the ReadIndex fence: quorum-confirmed, never stale, one heartbeat round-trip per read. |
replicated |
The write is applied to the local engine first, then handed to the replication engine. The -consistency flag decides when the client is ACKed. |
Primary-copy durability across N copies; NOT linearizable under concurrent writers (no single-writer ordering). Reads are local. |
The --consistency flag (replicated mode)
-consistency only applies in replicated mode and controls the point at which the client's write is acknowledged:
| Value | ACK semantics | Error behavior |
|---|---|---|
eventual default | Async — ACKs immediately after the local write; replicas catch up in the background. | No blocking wait; a replica that is down simply falls behind. |
quorum | ACKs only after a majority of replicas (counting the local copy) have applied the write. | Surfaces ErrQuorumNotReached if a majority cannot be reached. |
strong | ACKs only after all replicas have applied the write. | Surfaces ErrReplicationTimeout (or ErrQuorumNotReached) if the required copies cannot be reached — for example, a strong write with a majority of replicas down errors instead of silently ACKing. |
Linearizable reads: --linearizable-reads (raft mode)
By default, raft-mode reads are served from whatever state the receiving node has locally applied — fast, but possibly stale (a partitioned ex-leader can serve reads after a new leader has committed writes elsewhere). Starting --linearizable-reads (default false) closes that hole by running every GET through the Raft §6.4 ReadIndex fence, without writing anything to the log:
- Capture the fence — the leader records its current
commitIndexas the read fence. The fence is only valid once the leader has committed an entry in its own term; immediately after an election it returns a retryable "leadership not yet established" error until the election no-op commits. - Quorum heartbeat round — the leader sends a fresh heartbeat to its peers and waits for a quorum of acknowledgements, proving no higher-term leader existed at that moment.
- Apply catch-up — the read waits until the local state machine has applied up to the fence, then reads from the engine.
Semantics and cost: reads are quorum-confirmed and never stale, at the price of one heartbeat round-trip per GET (plus any apply catch-up). Only the leader can serve them — followers reject reads with a MOVED <leader-addr> redirect, exactly like writes, which the cluster-aware client follows transparently. Each fence is bounded by a 2-second timeout covering the heartbeat round and apply catch-up. The flag has no effect outside --mode=raft (the server logs a warning and ignores it).
Automatic rebalancing: --auto-rebalance
In distributed modes (v1.1.0+), membership events — node join, leave, or failure — no longer just update routing: they trigger a ring rebalance plus physical key migration. A rebalancer subscribes to partition-map membership events, debounces bursts of gossip (3-second window), then recomputes partition ownership and hands off to the TransferAgent, which streams re-owned keys to their new owners over a dedicated partition-transfer HTTP listener on client port +5 (override with -transfer-addr). Local copies are deleted only after the destination confirms receipt, so an interrupted migration is simply retried on the next membership event — it re-sends the unconfirmed remainder. When the cluster TLS flags are set, transfer traffic uses the same inter-node TLS/mTLS configuration.
Auto-rebalance is on by default; disable it with --auto-rebalance=false if you want membership changes to update routing only, with data migration under manual control (the transfer listener still starts so other nodes can push keys to this one).
Rack-aware placement (v1.1.0): start each node with --rack-id <zone> (e.g. --rack-id=asia-south1-a) and rebalance never places two copies of a partition in the same failure domain — unless the replication factor exceeds the number of distinct racks, in which case it falls back to distinct nodes while still spanning every rack available. Rack labels spread between members via gossip digests, so only the local node needs the flag, and /admin/cluster reports each node's "rack". Rackless clusters keep the exact previous deterministic round-robin placement. See Cluster & Consensus.
Launching a 3-node Raft cluster
Run the same command on each host with an identical -peers value, changing only -node-id and -addr:
# 3-node raft cluster (run on each host, same --peers value)
veltrixdb -mode raft -node-id n1 -addr 127.0.0.1:9000 \
-peers n1@127.0.0.1:9000,n2@127.0.0.1:9100,n3@127.0.0.1:9200Repeat on the other two hosts with -node-id n2 / -addr 127.0.0.1:9100 and -node-id n3 / -addr 127.0.0.1:9200 respectively. Each peer entry is id@host:port, where host:port is that peer's own -addr (its client-facing storage port) — inter-node Raft, replication, gossip, and partition-transfer listeners are derived automatically from that port (by default, client port +2 / +1 / +3 / +5 respectively) unless overridden with -raft-addr, -repl-addr, -gossip-addr, or -transfer-addr.
Cluster-aware client behavior
The bundled client (client.NewClient in Go; equivalent logic ships in each SDK) is a real cluster client, not a dumb connection wrapper:
- Topology discovery — it fetches cluster topology over the storage port using the
TOPOLOGYcommand (mirrored at the HTTP endpoint/admin/cluster), which reports role, Raft term/leader, peers, partition epoch, and per-replica replication lag. - Consistent-hash routing — it builds a consistent-hash ring identical to the server's own (same hash function, 64 virtual nodes per physical node) and routes each key directly to the node that owns it, avoiding an extra network hop through a random node.
- MOVED redirects — if a request lands on a non-leader in
raftmode, the server rejects it withMOVED <addr> <leader-id>(orMOVED -if the leader is currently unknown); the client transparently retries against the redirected leader, following further hops up to a bounded limit.
Known limitations
raft mode gives linearizable writes via quorum commit, but by default reads are served from whatever state has been locally applied on the node handling the request — a client that reads immediately after a write acknowledged by a different node may observe stale data. Start the cluster with --linearizable-reads when you need never-stale reads, and budget one heartbeat round-trip per GET.replicated mode is primary-copy replication built for durability across copies, not for ordering. It has no single-writer ordering, so two concurrent writers to the same key are not linearizable under any -consistency setting, including strong. Use raft mode when write linearizability matters.IDXCREATE/IDXDROP must be issued on each node; raft mode replicates it), and QUERY/RANGE/SCANCUR reads, which are always served from local state in cluster modes. Additionally, the cross-region shipper repl-ship has no back-pressure to the source and resolves conflicts last-write-wins only. See Data Model for what these operations do.Decision guidance
| If you need… | Use mode |
|---|---|
| Maximum throughput, single node, no HA requirement | standalone |
| Linearizable writes with automatic failover across a quorum | raft |
| Strongly consistent reads with no staleness window | raft with --linearizable-reads — every GET is quorum-confirmed via the ReadIndex fence, at one heartbeat round-trip per read (leader-only; followers redirect). |
| Durability across multiple copies with the lowest write latency, and can tolerate eventual convergence | replicated with -consistency eventual |
| Durability across a majority of copies before acknowledging a write, without requiring all replicas up | replicated with -consistency quorum |
| Every replica confirmed before ACK, and can tolerate rejected writes when a replica is down | replicated with -consistency strong |
| Namespace / hash-field / vector / list / set / secondary-index writes to be cluster-wide | raft or replicated — routed through the coordinator in both since v1.1.0 (exception: replicated-mode IDXCREATE/IDXDROP metadata stays node-local, and QUERY/RANGE/SCANCUR reads are always local). |