Replication & CDC

VeltrixDB exposes an in-process change data capture (CDC) stream over HTTP, and ships a standalone repl-ship binary that tails it to forward writes to a remote cluster for cross-region or cross-cluster replication.

In-process CDC

Every applied write or delete is published to an in-process CDC stream, exposed as newline-delimited JSON (NDJSON) over HTTP:

curl http://localhost:2112/admin/cdc?prefix=

The optional prefix query parameter filters the stream to keys starting with that prefix; an empty prefix streams every event. Each line is a JSON object describing the operation:

{"Op":"PUT","Key":"orders/4521","Value":"...","Timestamp":1751464805123456789}
{"Op":"DEL","Key":"orders/4400","Timestamp":1751464806012345678}

This endpoint is a live subscription — the connection stays open and streams events as they happen, similar to tailing a log.

Cross-process forwarding with repl-ship

The repl-ship binary subscribes to a local node's /admin/cdc endpoint and forwards each event to a remote VeltrixDB cluster over its binary protocol — PUT events become MultiPut calls, DEL events become deletes:

go build -o repl-ship ./cmd/repl-ship

repl-ship --src http://primary:2112 \
  --dst-tcp replica.us-east:9000 \
  --batch 64 \
  --prefix "" \
  --deadletter /var/log/repl-ship/deadletter.jsonl

When the source node's admin API is guarded (-admin-token — see Admin API access control), pass the same token to repl-ship via --src-token or the VELTRIX_ADMIN_TOKEN environment variable so it can authenticate against /admin/cdc and /admin/changes.

repl-ship maintains one persistent TCP connection to the destination and reconnects on error. On remote-write failure it retries with exponential backoff up to --max-retries, then appends the event to the deadletter file for manual replay. Conflict resolution beyond last-write-wins relies on the remote cluster's own version-vector deduplication — repl-ship does not resolve conflicts itself.

Durable cross-region catch-up

The live CDC stream is in-memory: a subscriber that is down misses every event published while it was away. The /admin/changes feed closes that gap. It is an index-backed catch-up feed — it scans the index for entries written at or after a microsecond cursor, so a restarted shipper can replay exactly the delta it missed before rejoining the live stream:

curl "http://localhost:2112/admin/changes?since=1751464805123456&limit=10000"

The response is NDJSON: CDC-shaped events ordered by write timestamp, followed by one trailer line with the pagination cursor:

{"Op":"PUT","Key":"orders/4521","Value":"...","Timestamp":1751464805123456}
{"Op":"DEL","Key":"orders/4400","Timestamp":1751464806012345}
{"cursor":1751464806012345,"more":false}

Semantics of the feed:

• Tombstones are included (Op:"DEL") — deletes made during shipper downtime reach the remote region too. Expired-TTL entries are skipped (they expire remotely on their own).
• The since boundary is inclusive (>=): events sharing the checkpoint's exact microsecond re-ship rather than being lost. Destination applies are last-write-wins puts/deletes, so replays are harmless.
• Pagination: when more is true in the trailer, resume from the returned cursor (the timestamp of the last event on the page — inclusive, per the point above). limit defaults to 10,000 and is capped at 100,000.

To make repl-ship use the feed, run it with a checkpoint file:

repl-ship --src http://primary:2112 \
  --dst-tcp replica.us-east:9000 \
  --checkpoint /var/lib/repl-ship/ckpt \
  --checkpoint-every 256

repl-ship persists the last shipped write-timestamp (µs) to the checkpoint file — atomically, via tmp-file + rename — after every --checkpoint-every shipped events and on shutdown. On restart it first pages through /admin/changes from the checkpoint cursor, shipping the entire missed delta, and only then subscribes to the live /admin/cdc stream — zero event loss across shipper downtime, at last-write-wins semantics. Without --checkpoint, no catch-up runs on restart and events published while the shipper was down are still missed.

Treat repl-ship-based replication as asynchronous, eventually-consistent geo-distribution. For durability across copies within a cluster, use the Raft or replication-engine consistency modes described below, and use backup/PITR for point-in-time recoverability.

Replication consistency modes

CDC and repl-ship are one of two replication mechanisms in VeltrixDB. The other — Raft consensus for linearizable writes, and the primary-copy replication engine with eventual/quorum/strong consistency levels — operates independently of CDC and is the primary durability and consistency mechanism for clustered deployments. Those modes, their guarantees, and their trade-offs are covered in full in Cluster & Consensus; this page only covers the CDC/forwarding path.