Replication & CDC
VeltrixDB exposes an in-process change data capture (CDC) stream over HTTP, a durable catch-up feed, and a standalone repl-ship binary that forwards writes to a remote cluster. In-cluster replication (Raft or the replication engine) is a separate mechanism, summarised at the end.
In-process CDC
Writes and deletes are published to an in-process CDC stream (in v1.12 only single-key ones — see below), exposed as newline-delimited JSON (NDJSON) over HTTP on the admin port (loopback-only unless -admin-token is set):
curl "http://localhost:2112/admin/cdc?prefix=&buffer=1024"The optional prefix query parameter filters the stream to keys starting with that prefix; an empty prefix streams every event. buffer sets this subscription's channel depth (default 1024). Each line is a JSON object describing the operation:
{"Op":"PUT","Key":"orders/4521","Value":"...","Timestamp":1751464805123456}
{"Op":"DEL","Key":"orders/4400","Timestamp":1751464806012345}Timestamp is microseconds since the Unix epoch, and Value is base64 (JSON encoding of the raw bytes). This endpoint is a live subscription — the connection stays open and streams events as they happen, similar to tailing a log.
PUT and DEL. Batched writes — MPUT, pipelined PUTs the server coalesces into one batch, transactions, Raft's coalesced apply — and atomic ops (CAS/INCR/DECR/SETNX) produce no live event; their final state is visible only through the /admin/changes feed. The broker never blocks the writer: when a subscriber's channel is full the event is dropped for that subscriber (cdc_dropped_total in /admin/stats, not a Prometheus metric), and after 3 drops in total over its lifetime (not consecutive) the subscription is closed. Fixed in v1.14 every write path emits one event per key — PUT, DEL, MPUT, pipelined PUTs, transactions, Raft batch apply, and the atomic ops (as PUT; a CAS mismatch or a SETNX on an existing key writes nothing and emits nothing). TTL expiry emits nothing. A subscriber is closed only after 3 consecutive drops; a successful send resets the count.
Cross-process forwarding with repl-ship
The repl-ship binary subscribes to a local node's /admin/cdc endpoint and forwards each event to a remote VeltrixDB cluster over its binary protocol — each PUT event becomes a binary PUT (0x01) and each DEL event a binary DEL (0x03), sent one at a time. It is not in the server image; build it from the main repository:
go build -o repl-ship ./cmd/repl-ship
repl-ship --src http://primary:2112 \
--dst-tcp replica.us-east:9000 \
--batch 64 \
--prefix "" \
--deadletter /var/log/repl-ship/deadletter.jsonl| Flag | Default | Purpose |
|---|---|---|
--src | http://127.0.0.1:2112 | Local VeltrixDB metrics/admin URL (CDC source) |
--dst-tcp | — | Remote VeltrixDB binary-protocol address (required) |
--batch | 64 | Events buffered before a flush (a flush also runs every 200 ms). The flag help says "per remote MultiPut", but events in a flush are still sent as individual binary PUT/DEL requests. |
--max-retries | 5 | Retry budget per event (exponential backoff) before deadlettering |
--deadletter | /var/log/repl-ship/deadletter.jsonl | Path for events that exhaust their retry budget |
--prefix | empty | CDC key-prefix filter; empty ships everything |
--checkpoint | empty | Path to the checkpoint file (last shipped write-timestamp). Empty = no catch-up on restart |
--checkpoint-every | 256 | Persist the checkpoint after this many shipped events |
--src-token | empty (falls back to VELTRIX_ADMIN_TOKEN) | Admin bearer token sent to the source's /admin/cdc and /admin/changes when the source runs with -admin-token. Fixed in v1.14 --admin-token is accepted as an alias, the CDC prefix is URL-escaped and HTTP error bodies are logged |
When the source node's admin API is guarded (-admin-token — see Admin API access control), pass the same token to repl-ship via --src-token or the VELTRIX_ADMIN_TOKEN environment variable so it can authenticate against /admin/cdc and /admin/changes.
repl-ship maintains one persistent TCP connection to the destination and reconnects on error. On remote-write failure it reconnects and retries with exponential backoff (100 ms, 200 ms, 400 ms, …) up to --max-retries, then appends the event to the deadletter file for manual replay. There is no conflict resolution: each event is applied at the destination as a plain PUT/DEL in arrival order (last write wins). The destination does no version-vector deduplication either — version vectors exist in the replication library but are not used.
Durable cross-region catch-up
The live CDC stream is in-memory: a subscriber that is down misses every event published while it was away. The /admin/changes feed closes that gap. It is an index-backed catch-up feed — it scans the index for entries written at or after a microsecond cursor, so a restarted shipper can replay exactly the delta it missed before rejoining the live stream:
curl "http://localhost:2112/admin/changes?since=1751464805123456&limit=10000"The response is NDJSON: CDC-shaped events ordered by write timestamp, followed by one trailer line with the pagination cursor:
{"Op":"PUT","Key":"orders/4521","Value":"...","Timestamp":1751464805123456}
{"Op":"DEL","Key":"orders/4400","Timestamp":1751464806012345}
{"cursor":1751464806012345,"more":false}Semantics of the feed:
- Tombstones are included (
Op:"DEL") — deletes made during shipper downtime reach the remote region too. Expired-TTL entries are skipped (they expire remotely on their own). - The
sinceboundary is inclusive (>=): events sharing the checkpoint's exact microsecond re-ship rather than being lost. Destination applies are last-write-wins puts/deletes, so replays are harmless. - Pagination: when
moreistruein the trailer, resume from the returnedcursor(the timestamp of the last event on the page — inclusive, per the point above).limitdefaults to 10,000; a value above 100,000 (or ≤ 0) falls back to 10,000. - Last state only: the feed is built from the index, so it returns each key's current value or tombstone, not intermediate versions. It covers batched writes and atomic ops, which the v1.12 live stream misses.
To make repl-ship use the feed, run it with a checkpoint file:
repl-ship --src http://primary:2112 \
--dst-tcp replica.us-east:9000 \
--checkpoint /var/lib/repl-ship/ckpt \
--checkpoint-every 256repl-ship persists the last shipped write-timestamp (µs) to the checkpoint file — atomically, via tmp-file + rename — after every --checkpoint-every shipped events and on shutdown. On restart it first pages through /admin/changes from the checkpoint cursor, shipping the entire missed delta, and only then subscribes to the live /admin/cdc stream — no lost final states across shipper downtime, at last-write-wins semantics (intermediate versions of a key are not replayed). Without --checkpoint, no catch-up runs on restart and events published while the shipper was down are still missed.
repl-ship is running it ships only what the live stream carries, so batched writes and atomic ops reach the destination only after a restart with --checkpoint replays /admin/changes (Fixed in v1.14 the live stream carries every write path). And there is no back-pressure signal to the source: local writes keep succeeding even while the destination is degraded or unreachable.
Treat repl-ship-based replication as asynchronous, eventually-consistent geo-distribution. For durability across copies within a cluster, use the Raft or replication-engine consistency modes described below, and use backup/PITR for point-in-time recoverability.
In-cluster replication modes
CDC and repl-ship are independent of in-cluster replication, which --mode selects. Raft and the replication engine are mutually exclusive: a server runs exactly one of them (in raft mode no replication engine exists). Full detail is in Cluster & Consensus.
--mode=raft | --mode=replicated | |
|---|---|---|
| Writes | Quorum-committed through one Raft log; non-leaders answer MOVED <addr> <id> | Any node writes locally, then ships to its peers (port client + 1) |
| Acknowledged after | Commit and apply on the leader | --consistency: eventual (default) = local write; quorum = RF/2+1 copies; strong = RF copies (10 s timeout) |
| Reads | Local applied state. GET and MGET pass a read barrier (a new leader waits for its no-op; --linearizable-reads adds the ReadIndex fence). In v1.12 namespace, hash and RANGE/SCANCUR reads skip it and can be stale; Fixed in v1.14 every read passes it | Always local |
| Conflicts | None (single log order). On v1.16 and earlier the SETIF version token is each node's own timestamp, so a guarded TXN can hold on one node and fail on another; Fixed on main after v1.16 — ships in the next release versions come from the leader's clock and match on every node (details) | None resolved: replicas apply ops in arrival order. Version vectors are defined but never set or compared, and in v1.12 veltrixdb_replication_conflict_resolutions_total stays 0 (removed in v1.14, with vector_clock_updates_total) |
| Catch-up | AppendEntries backfill (on v1.16 and earlier a restarted node re-applies its retained log, doubling INCRs; Fixed on main after v1.16 — ships in the next release exactly once, details); InstallSnapshot once the log passed 8192 entries (on v1.16 and earlier: snapshot capped at 256 MB, TTLs not preserved, deleted keys survive on the follower; Fixed on main after v1.16 — ships in the next release no cap, streamed in chunks, restore equals the snapshot, TTLs kept — details) | Anti-entropy every 30 s re-sends un-acked ops to LAG replicas only. Fixed in v1.14 FAILED replicas are re-probed and caught up (below) |
FAILED; failed replicas are skipped by replication and by anti-entropy, and nothing moves them back to SYNC — in practice a replica stays failed until the primary restarts. A full replication write queue (10,000 ops) rejects the op. Watch /admin/cluster (replication[].state) and veltrixdb_replication_failures_total. Replication lag is exported as veltrixdb_replication_lag_bytes; veltrixdb_replication_lag_nanoseconds is always 0.Fixed in v1.14 the recovery worker re-probes each
FAILED replica with backoff (1 s, doubling to 30 s); one that answers goes to SYNC_PENDING, is sent every write it has not acked (or, if the in-memory retention buffer overflowed, the primary’s change feed first) and returns to SYNC. A full write queue keeps the op and lets anti-entropy deliver it. veltrixdb_replication_lag_nanoseconds becomes the age of the oldest write a replica has not acked; new gauges replicas_failed / replicas_sync_pending and counters recovery_probes_total / replica_recoveries_total. Tombstones are reaped only once every replica’s watermark has passed them (capped at 2× the grace period). Retention is in memory: on v1.16 and earlier a replica that was down when its primary restarted does not get the writes made before that restart (see below).
Replica catch-up across a primary restart Fixed on main after v1.16 — ships in the next release
On v1.16 and earlier the retention buffer and per-replica acks are in memory only, so a restarted primary takes every replica as caught up: a replica that was down while its primary restarted never receives the writes and deletes made before the restart. A clean shutdown also keeps no tombstones, and a returning node can push its stale copies of peers' keys back and undo their deletes. On main:
- Each replica's acknowledgement watermark is persisted in
<--data>/replication_state.json(at most once per second, never per write). At startup a replica with a persisted watermark startsFAILEDwith a pending catch-up and returns toSYNConly after it was caught up from the change feed, deletes included. - A clean shutdown's checkpoint keeps the tombstones a replica still needs. Writes applied from a peer are marked and never re-sent, so a returning node no longer undoes its peers' deletes.
- A replica behind by more than the tombstone retention (2 × the grace period, 48 h by default) is fully resynced and flagged with an ERROR log line
FULL RESYNC: a delete whose tombstone was already reaped cannot be replayed, so the key may survive on it — rebuild that replica from a healthy node.GET /admin/clustershowsresync_pendinganddelete_gapper replica; the metrics are on Monitoring. - Upgrade: the first start of the new build has no state file and takes every replica as caught up, as before — drain a v1.16 node before stopping it for the upgrade.
Still open: concurrent writers to one key on different nodes are not ordered (no conflict resolution), so the nodes can end up holding different values for it permanently. Write each key through one node if replicas must agree, or use raft mode.
-net=cpp|uring|poll front-end writes straight to the local engine, bypassing Raft and replication routing, so the server refuses to start with it in -mode raft or -mode replicated. Clustered nodes must use the default -net=go. A standalone destination running the C++ front-end can still receive repl-ship traffic, since it serves binary PUT and DEL.