Backup & Restore

VeltrixDB backs up the write-ahead log and Value Log directly, supports full and incremental backup chains (online through the admin API), uploads to S3/GCS/Azure, adds point-in-time recovery from continuous WAL archiving, and restores offline onto fresh data directories.

What gets backed up

VeltrixDB stores two files per disk: wal.log (a compacted write-ahead log — one entry per live key, holding the key and its VLog offset) and vlog_active.dat (the append-only value bytes). Backup captures both. The in-memory index is not backed up — it is rebuilt from the WAL on startup, so backing up the WAL and VLog is sufficient to reconstruct full key-value state.

WAL encoding. New in v1.6 The backed-up wal.log and PITR archive segments use the binary WAL format (48-byte header, magic 0xB1, CRC32C over the whole record): the veltrixdb-backup CLI, which opens the engine itself, always writes binary, and restore-pitr always appends binary records. Any build that reads binary also reads legacy text records, so restoring into the same or a newer build always works. A pre-binary build (v1.5 and earlier) cannot read binary records: to restore into one, restore with the current build, start the server once with --wal-format=text and stop it cleanly (the shutdown checkpoint rewrites wal.log as text), then start the older build. Keys containing | or a newline are not crash-safe in the text format.

Fixed in v1.14 backups, checkpoints and PITR archive records carry each key’s absolute TTL expiry (v1.12 drops TTLs, so restored keys never expire), so a restored key expires when the original would have. A build older than that change cannot read the record of a TTL’d key: to roll back to one, start the new build once with --wal-format=text-legacy (not text) and stop it cleanly; TTL’d keys become immortal on the older build.

Full backup

A full backup captures a consistent snapshot of the entire dataset as of the moment it runs, while the engine keeps serving traffic (on v1.16 and earlier the writes that follow are not crash-safe until a clean restart — see above). It checkpoints the WAL, then copies each disk's VLog up to its snapshotted end offset (through its own read-only descriptor, so it never reads the region concurrent appends are writing) plus the compacted WAL, and writes manifest.json last.

Against a running server, use the admin API. POST /admin/backup (loopback-only unless -admin-token is set) makes the server itself write the backup, so dest_dir is a path on the server's filesystem:

# Full backup of a RUNNING server
curl -s -X POST localhost:2112/admin/backup \
  -d '{"type":"full","dest_dir":"/backups/full-20260523"}'
# same thing via the CLI (v1.12 sends no admin token; v1.14 adds --admin-token / VELTRIX_ADMIN_TOKEN)
veltrix --addr 127.0.0.1:2112 backup /backups/full-20260523

The response is JSON with status, type, backup_id, dest_dir, num_disks, duration_ms and the full manifest.

go build -o veltrixdb-backup ./cmd/backup

# Full backup of a STOPPED node's data dirs
veltrixdb-backup full --data-dirs=/mnt/nvme0,/mnt/nvme1 --dest=/backup/2026-05-26

# Full backup + upload in one step (stages in a temp dir; server stopped)
veltrixdb-backup full-cloud --data-dirs=/mnt/nvme0,/mnt/nvme1 \
  --provider=s3 --bucket=my-bucket --region=us-east-1

full, incremental and full-cloud accept --cache-mb (default 64) for the engine they open.

Incremental backup

An incremental backup captures only the VLog bytes written since a base backup, plus a fresh copy of the (small) compacted WAL, which always reflects every live key at that moment:

# Running server: base_dir must hold the base backup's manifest.json
curl -s -X POST localhost:2112/admin/backup \
  -d '{"type":"incremental","dest_dir":"/backup/incr-20260523-150000","base_dir":"/backup/full-20260523"}'

# Stopped server only
veltrixdb-backup incremental --data-dirs=/mnt/nvme0,/mnt/nvme1 \
  --dest=/backup/incr-20260523-150000 --base=/backup/full-20260523

VLog offsets never change: GC relocates a live value by appending a new copy at the tail (and later punches holes in the dead head of the file), so each incremental delta is a self-contained, ordered byte range. A backup chain looks like:

full-T1  <-  incr-T2  <-  incr-T3  <-  incr-T4

Restoring to T4 requires every backup in the chain; VLog deltas are concatenated in order (full_vlog + delta_T2 + delta_T3 + delta_T4) and the WAL is taken from the latest backup in the chain, since it already reflects all live keys at that point.

Cloud upload

The backup tool uploads to S3, GCS, or Azure with the same manifest format used locally:

# Upload an existing local backup (prints the cloud path: <prefix>/<backup_id>/)
veltrixdb-backup upload --src=/backup/2026-05-26 \
  --provider=s3 --bucket=my-bucket --region=us-east-1

# Download a cloud backup (--cloud-path is what upload printed)
veltrixdb-backup download --provider=s3 --bucket=my-bucket --region=us-east-1 \
  --cloud-path=veltrixdb-backups/full-1748001600000000000/ --dest=/tmp/restore

# List everything in cloud storage
veltrixdb-backup list-cloud --provider=s3 --bucket=my-bucket --region=us-east-1

Every cloud command takes --provider=s3|gcs|azure, --bucket (S3/GCS bucket or Azure container) and --prefix (default veltrixdb-backups); backups land under <prefix>/<backup_id>/, with backup_id taken from manifest.json. list-cloud prints cloud path, backup ID and size. Credentials can be passed as flags (--aws-access-key, --aws-secret-key, --region, --gcs-token, --gcs-cred-file, --azure-account, --azure-key), but prefer the provider's standard environment variables, which are used when the flags are empty (flags end up in shell history and process listings): AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION / AWS_SESSION_TOKEN for S3, GOOGLE_APPLICATION_CREDENTIALS (or GCS_ACCESS_TOKEN) for GCS, and AZURE_STORAGE_ACCOUNT / AZURE_STORAGE_KEY / AZURE_STORAGE_CONTAINER for Azure.

Restore

# Restore from a local backup chain (oldest-first; one --data-dirs entry per disk)
veltrixdb-backup restore \
  --chain=/backups/full-20260523,/backups/incr-20260523-150000 \
  --data-dirs=/mnt/nvme0,/mnt/nvme1

# Download from cloud, then restore
veltrixdb-backup download --provider=s3 --bucket=my-bucket --region=us-east-1 \
  --cloud-path=veltrixdb-backups/full-1748001600000000000/ --dest=/backups/full
veltrixdb-backup restore --chain=/backups/full --data-dirs=/data-new

restore does not check that the target directories are empty: it overwrites vlog_active.dat and wal.log in them (restore-pitr, below, refuses non-empty targets).

Restore copies the full backup's VLog into the destination, appends each incremental delta in order, and copies the WAL from the latest backup in the chain onto the destination. Once restore completes, start the engine normally — replayWAL() rebuilds the in-memory index from the restored WAL, and the engine begins serving traffic with a cold cache but correct data.

Point-in-time recovery (PITR)

For recovery finer than backup granularity, VeltrixDB supports continuous WAL archiving, enabled on the server with -archive-dir (plus -archive-interval-ms, default 1000, and the retention flags -archive-max-age-sec / -archive-max-bytes, 0 = unlimited). A background WALArchiver copies newly durable WAL entries (with their value bytes resolved inline, so each archived segment is self-contained and safe from VLog GC) into an archive directory as they're fsynced. Enable archiving before taking the base full backup, since restore needs archive coverage from the backup point forward:

# Inspect what's in the archive
veltrixdb-backup archive-status --archive=/backup/wal-archive

# Restore to an exact wall-clock moment
veltrixdb-backup restore-pitr \
  --base-backup=/backup/full-20260702 \
  --archive=/backup/wal-archive \
  --until=2026-07-02T14:30:00Z \
  --data=/data-new

# Restore to an exact single write
veltrixdb-backup restore-pitr \
  --base-backup=/backup/full-20260702 \
  --archive=/backup/wal-archive \
  --until=version:123456 \
  --data=/data-new

restore-pitr requires the engine to be stopped and the target --data directories to be fresh and empty — it refuses to touch a directory that already contains engine files. Archive segments are verified with per-segment CRC32C before replay; a corrupted segment aborts the restore rather than silently applying a torn record. Binary records also carry their own CRC32C, checked on decode. The archiver, PITR restore and crash replay all decode WAL bytes with the same reader, which accepts binary and legacy text records mixed.

The archive is laid out per disk as <archive-dir>/disk<N>/seg-<seq>.wal with a seg-<seq>.json sidecar (version range, first/last entry time in UnixNano, entry count, size, CRC32C). restore-pitr appends the selected records to the restored wal.log; the next engine start replays them.

Recommended backup strategy

A common pattern is a daily full backup plus hourly incrementals, which lets you restore to any hour by applying the full plus N incrementals. For finer recovery, run continuous WAL archiving alongside a daily full and use restore-pitr to land on an exact timestamp or write version.

Retention guidance: keep at least 7 daily full backups plus all incrementals for the latest full — older incrementals are useless once their base full is deleted. Size the archive's retention (-archive-max-age-sec / -archive-max-bytes) so it always spans back to the oldest full backup you intend to restore from.

After every backup run, verify (and periodically restore to staging — a manifest alone does not prove the data is readable): manifest.json exists and its engine_version matches the live engine, vlog_end_off per disk has increased, and each disk's wal.log size is proportional to the number of live keys.

What is not backed up