Deployment Modes

VeltrixDB serves every request from local state by default. The distributed layer is opt-in via the -mode flag and is wired directly into the serving path: every mutating request — plain KV (PUT/DELETE/MultiPut), atomics, TXN, and (since v1.1.0) namespace, hash-field, vector, secondary-index, and list/set writes — passes through a write coordinator in front of the storage engine, on both the text and binary protocols.

The three modes

Set the mode with -mode standalone|raft|replicated; raft and replicated are mutually exclusive (one coordinator per process, never both). Here is exactly what each mode guarantees for writes and reads — this is the load-bearing reference for choosing a mode, so read it precisely rather than skimming.

The --consistency flag (replicated mode)

-consistency only applies in replicated mode and controls the point at which the client's write is acknowledged:

Linearizable reads: --linearizable-reads (raft mode)

By default, raft-mode reads are served from whatever state the receiving node has locally applied — fast, but possibly stale (a follower may lag; a partitioned ex-leader can serve reads after a new leader has committed writes elsewhere). New in v1.12 A freshly elected leader waits for its term's no-op to apply before serving GET / MGET (WaitLeaderApplied, ≤ 2 s), so reads on the leader include every write its predecessor acknowledged. Starting --linearizable-reads (default false) closes the remaining hole by running every GET and MGET through the Raft §6.4 ReadIndex fence, without writing anything to the log:

1. Capture the fence — the leader records its current commitIndex as the read fence. The fence is only valid once the leader has committed an entry in its own term; immediately after an election it returns a retryable "leadership not yet established" error until the election no-op commits.
2. Quorum heartbeat round — the leader sends a fresh heartbeat to its peers and waits for a quorum of acknowledgements, proving no higher-term leader existed at that moment.
3. Apply catch-up — the read waits until the local state machine has applied up to the fence, then reads from the engine.

Semantics and cost: reads are quorum-confirmed and never stale, at the price of one heartbeat round-trip per read (plus any apply catch-up). Only the leader can serve them — followers reject reads with a MOVED <leader-addr> <leader-id> redirect, exactly like writes, which the cluster-aware client follows transparently. Each fence is bounded by a 2-second timeout covering the heartbeat round and apply catch-up. The flag has no effect outside --mode=raft (the server logs a warning and ignores it).

Automatic rebalancing: --auto-rebalance

In replicated mode, membership events — a node added to or removed from the membership — trigger a ring rebalance plus physical key migration. Node state changes (failed, recovering, active) never start a migration, and in raft mode migration never runs at all (every node holds the full state; -auto-rebalance is a logged no-op there). On v1.14 and earlier a state change also started a migration, which could delete committed keys — run those releases with -auto-rebalance=false. A rebalancer subscribes to partition-map membership events, debounces bursts of gossip (3-second window), then recomputes partition ownership and hands off to the TransferAgent, which streams re-owned keys to their new owners over a dedicated partition-transfer HTTP listener on client port +5 (override with -transfer-addr). Local copies are deleted only after the destination confirms receipt, so an interrupted migration is simply retried on the next membership event — it re-sends the unconfirmed remainder. Each batch carries the sender's membership epoch and a receiver with a newer one refuses it (HTTP 409). Vector-namespace settings (@vecns/) and index definitions (@idxdef/) are copied to every node rather than moved. When the cluster TLS flags are set, transfer traffic uses the same inter-node TLS/mTLS configuration.

On v1.16 and earlier membership is static. Each server registers the nodes in --peers (or --seeds) at startup; there is no runtime join or drain command and gossip does not carry membership, so adding a node means restarting every node with it in --peers. New on main after v1.16 — ships in the next release nodes are added and removed online — see the membership runbook below; in replicated mode an online add or remove is what triggers the migration.

Signing node-to-node traffic: --cluster-secret-file New in v1.11

The transfer listener carries both key migration (/transfer/keys) and distributed search (/internal/search). Put the same secret (at least 16 bytes) on every node with --cluster-secret-file or VELTRIXDB_CLUSTER_SECRET: every request is then signed with HMAC-SHA256 over time, method, path and body, and refused (HTTP 401) if it does not verify or is more than 5 minutes off the receiver's clock. Without a secret or --cluster-mtls, the listener is unauthenticated and the server logs a warning at startup.

Search in cluster modes

Vector, text and hybrid searches, QUERY and IDXQUERY run on every non-failed node and are merged (--search-fanout, default true). The request fails closed: a peer that misses --search-timeout-ms (2000) fails it with an error naming the peer, unless --search-allow-partial is set. See Cluster & Consensus and Vector & Hybrid Search.

Auto-rebalance is on by default; disable it with --auto-rebalance=false if you want membership changes to update routing only, with data migration under manual control (the transfer listener still starts so other nodes can push keys to this one). Fixed in v1.15 it has no effect in --mode=raft.

Rack-aware placement (v1.1.0): start each node with --rack-id <zone> (e.g. --rack-id=asia-south1-a) and rebalance never places two copies of a partition in the same failure domain — unless the replication factor exceeds the number of distinct racks, in which case it falls back to distinct nodes while still spanning every rack available. Rack labels spread between members via gossip digests, so only the local node needs the flag, and /admin/cluster reports each node's "rack". Rackless clusters keep the exact previous deterministic round-robin placement. See Cluster & Consensus.

Launching a 3-node Raft cluster

Run the same command on each host with an identical -peers value, changing only -node-id and -addr:

# 3-node raft cluster (run on each host, same --peers value)
veltrixdb -mode raft -node-id n1 -addr 127.0.0.1:9000 \
  -peers n1@127.0.0.1:9000,n2@127.0.0.1:9100,n3@127.0.0.1:9200

Repeat on the other two hosts with -node-id n2 / -addr 127.0.0.1:9100 and -node-id n3 / -addr 127.0.0.1:9200 respectively. Each peer entry is id@host:port, where host:port is that peer's own -addr (its client-facing storage port) — inter-node Raft, replication, gossip, and partition-transfer listeners are derived automatically from that port (by default, client port +2 / +1 / +3 / +5 respectively) unless overridden with -raft-addr, -repl-addr, -gossip-addr, or -transfer-addr.

Adding and removing nodes online New on main after v1.16 — ships in the next release

On v1.16 and earlier the only way to change membership is to restart every node with a new --peers list. On main, every membership endpoint is on the admin listener with the same auth as the rest of /admin/* (--admin-token, else loopback only):

Answers: 200 done; 202 raft learner added but not promoted yet (poll members); 400 bad request or last voter; 404 unknown id; 409 duplicate id, a change already in progress (retry), a concurrent replicated change, or a failed drain; 503 no reachable quorum, no leader or coordinator unreachable. Any node accepts a request and forwards it. Forwarding, --join and every replicated-mode change are node-to-node traffic and need --cluster-secret-file (or --cluster-mtls) on every node; without one, a raft change still works when sent to the leader's admin endpoint, nothing else does. Change one node at a time.

Raft: add a node

1. Start the new node empty with --join:

   veltrixdb --mode=raft --node-id=n4 --addr=10.0.0.14:9000 --data=/data \
     --join=10.0.0.11:9000,10.0.0.12:9000 --cluster-secret-file=/etc/veltrix/secret

   (add --advertise-addr=host:port when --addr has no host, e.g. :9000). It never campaigns or votes until a leader adds it, asks the listed members to add it, and retries every 2 s.
2. The leader adds it as a learner (log entries or a chunked InstallSnapshot; never counted toward a quorum), then promotes it to voter once a catch-up round ends within 1 s.
3. Without a cluster secret, add it explicitly: veltrix --addr <leader-admin> cluster add-node n4@10.0.0.14:9000.
4. Check: veltrix cluster members on any node lists n4 voter. Raft mode moves no keys: the new node gets everything through the log.

Raft: remove a node

veltrix cluster remove-node n2 on any node. Nothing is migrated; stop the removed node and wipe its data before it is ever re-added. Removing the leader is allowed: it steps down once the removal commits (one election, about 0.5–1.5 s). The change is refused with 503 when the remaining voters would not form a quorum the leader recently heard from, and with 400 for the last voter. A removed node that keeps running cannot disrupt the cluster: members refuse its pre-vote, so it never bumps the term.

Replicated: add a node

1. Start it empty: --mode=replicated --node-id=r4 --addr=10.0.0.24:9000 --join=10.0.0.21:9000 --cluster-secret-file=… --consistency=<same>. Until it is added it is joining and refuses writes.
2. The coordinator (the member with the lowest ID) writes the next membership version and pushes it to every member, which persists it in <--data>/cluster_membership.json. Each member catches r4 up with every write it originated, then live batches, and copies every key it holds to r4 (put-if-absent, so a copy never overwrites a newer write or resurrects a delete).
3. Done when every member shows r4 SYNC (veltrix replication, resync_pending=false) and logged [transfer] migration done. Each member sends every key it holds.

Replicated: remove a node

veltrix cluster remove-node r2: r2 is drained first — it refuses new writes until every write it originated is acknowledged by every replica (default wait 2 min, --wait-ms); if that does not happen nothing is removed (409) and r2 serves writes again. Then the new membership is pushed, and r2 sends every key it holds to the remaining members, deleting each only after all of them acknowledged it. Move clients off r2 first; while another member is down nothing can be drained. A dead node cannot be drained: --force removes it without the drain, and writes it acknowledged but never replicated (possible with --consistency=eventual) are lost.

What wins on restart

Once any change was made online, the persisted membership (raft: configuration entries in the log and snapshot; replicated: cluster_membership.json) wins over --peers on every restart. A cluster that never changed membership online keeps following --peers. --join applies only to a node with no persisted membership that is not named in --peers, so one pod template can carry both flags.

Rolling upgrade and rollback

One node at a time; a node is done when it has caught up, not when its process is up. Stop with SIGTERM (a clean stop runs the WAL checkpoint), swap the binary, start on the same --data.

• Raft: upgrade the followers first, the leader last, and move to the next node only after the restarted one has applied a fresh write (write a marker key through the leader and read it back on the restarted node, or watch its veltrixdb_raft_applied_index). Fixed on main after v1.16 — ships in the next release when upgrading from v1.16, do not let a v1.16 follower lag past the leader's log compaction (an upgraded leader can only send it a single-shot snapshot that loses TTLs and does not apply deletes), and do not use SETIF / guarded TXN until every node is upgraded (see Cluster & Consensus).
• Replicated: wait until every replica shows SYNC on every node before the next one. Fixed on main after v1.16 — ships in the next release a peer persists each replica's acknowledgement watermark (<--data>/replication_state.json), so a node that was down while its peer restarted is still caught up. A node that still runs v1.16 has no such file: drain it (stop client writes to it and let its peers acknowledge them) before stopping it.
• Raft rollback: v1.14 and earlier cannot read raft_log.dat: wipe the node and re-sync. A node of the next release rolled back to v1.15 / v1.16 starts as it is only if it has no raft_snapshot.dat; otherwise wipe and re-sync.
• WAL rollback (every mode): v1.14 and later, v1.16 included, read the current WAL directly — no step. Only for v1.13 and older run once with --wal-format=text-legacy and stop cleanly first (TTL’d keys become immortal there); never use it for a v1.14+ target. See Disaster Recovery.

Cluster-aware client behavior

The bundled Go client in the server repo (client.NewClient) is a real cluster client, not a dumb connection wrapper. The other SDKs live in the separate Veltrixdb-client repository; check their own docs for cluster routing.

• Topology discovery — it fetches cluster topology over the storage port using the TOPOLOGY command (mirrored at the HTTP endpoint /admin/cluster), which reports role, Raft term/leader, peers, partition epoch, and per-replica replication lag.
• Consistent-hash routing — it builds a consistent-hash ring identical to the server's own (same FNV-1a + fmix64 hash, 64 virtual nodes per physical node) and routes each key directly to the node that owns it, avoiding an extra network hop through a random node.
• MOVED redirects — if a request lands on a non-leader in raft mode, the server rejects it with MOVED <addr> <leader-id> (or MOVED - if the leader is currently unknown); the client transparently retries against the redirected leader, following up to 3 hops per request.

Known limitations

Decision guidance