Disaster Recovery

A runbook-style summary of VeltrixDB's disaster recovery procedures, covering pod crash loops, data corruption, write outages, GC death-spirals, backup/restore, encryption key rotation, failed disks, metadata repair, WAL-format rollback and a lost raft quorum. Assumes kubectl admin access with pods labeled app.kubernetes.io/name=veltrixdb in the veltrixdb namespace (the kubectl veltrix plugin's defaults).

Severity classes

1. Pod crash loop (SEV-1)

Start by pulling logs and pod status:

kubectl logs -n veltrixdb POD --previous | tail -100
kubectl describe pod -n veltrixdb POD

2. Data corruption (SEV-0)

1. Confirm corruption via metrics:

   kubectl port-forward -n veltrixdb POD 2112:2112 &
   curl -s localhost:2112/metrics | grep -E 'scrub_corruption_total|vlog_gc_read_errors_total'

2. Find the bad disk: kubectl logs -n veltrixdb POD | grep -E '\[scrub\] disk=.*MISMATCH' (lines report MAGIC MISMATCH or CRC MISMATCH with the disk and offset)
3. Quarantine the pod: kubectl label pod -n veltrixdb POD veltrixdb.io/quarantine=true --overwrite
4. Wipe the node's data and restart — raft or replicated mode only, where the other nodes refill it (Raft log catch-up / InstallSnapshot). In standalone mode restore from backup instead (section 5):

   kubectl debug -n veltrixdb POD -it --image=busybox:1.36 --target=veltrixdb -- \
     sh -c 'rm -f /proc/1/root/mnt/nvme*/vlog_active.dat /proc/1/root/mnt/nvme*/wal.log*'
   kubectl delete pod -n veltrixdb POD

   In replicated mode only writes the primaries still hold un-acked are re-sent; there is no full-state resync, so verify the refilled node.
5. After replay completes, remove the quarantine label: kubectl label pod -n veltrixdb POD veltrixdb.io/quarantine- --overwrite
6. Root-cause the disk from a node debug pod: kubectl debug node/NODE -it --image=ubuntu:24.04 -- chroot /host smartctl -a /dev/nvme0n1

3. Write outage (SEV-0)

kubectl get pods -n veltrixdb -o wide
kubectl debug -n veltrixdb POD -it --image=busybox:1.36 --target=veltrixdb -- \
  sh -c 'df -h /proc/1/root/mnt/nvme*'
kubectl port-forward -n veltrixdb POD 2112:2112 &
curl -s localhost:2112/metrics | grep -E 'storage_wal_flushes_total|vlog_garbage_ratio'

4. GC death-spiral / garbage ratio > 65% (SEV-2)

1. Throttle writes immediately: kubectl veltrix quota-set tenant_42 1000 5000000, then watch GC work through it with kubectl veltrix gc-status (ratio per disk, runs, emergency state).
2. Do not look for a runtime GC-budget knob: the GC bandwidth tiers (60 MB/s throttled, 200 MB/s critical at ≥ 50% garbage, uncapped emergency mode at ≥ 65%) are compile-time constants in storage/defrag.go, and there is no environment variable that overrides them. -gc-threshold only sets when normal GC starts.
3. Long-term: add disks (scale volumeClaimTemplates, re-roll the StatefulSet).

5. Backup and restore

# Full backup of a RUNNING server: through the admin API, written on the
# server's own filesystem (dest_dir is a path inside the pod)
kubectl port-forward -n veltrixdb POD 2112:2112 &
curl -s -X POST localhost:2112/admin/backup \
  -d '{"type":"full","dest_dir":"/mnt/nvme0/backup/'$(date +%F)'"}'
# incremental: {"type":"incremental","dest_dir":"...","base_dir":"<previous backup dir>"}
# (same as a full backup: veltrix --addr 127.0.0.1:2112 backup DEST_DIR)

# Offline tool: ONLY while the server is STOPPED (see warning below)
veltrixdb-backup full --data-dirs=/mnt/nvme0,...,/mnt/nvme7 --dest=/backup/$(date +%F)

# Upload to S3 (prints the cloud path: <prefix>/<backup_id>/, default prefix veltrixdb-backups)
veltrixdb-backup upload --src=/backup/$(date +%F) \
  --provider=s3 --bucket=my-bucket --region=us-east-1

# Download and restore (stop the engine first)
veltrixdb-backup download --provider=s3 --bucket=my-bucket --region=us-east-1 \
  --cloud-path=veltrixdb-backups/full-1748001600000000000/ --dest=/tmp/restore
veltrixdb-backup restore --chain=/tmp/restore --data-dirs=/data-new

Kubernetes volume snapshots are an operator-side alternative if your CSI driver supports them: create a VolumeSnapshot object per PVC (there is no kubectl create volumesnapshot subcommand, and the main repository ships no snapshot manifests). Run POST /admin/checkpoint first so wal.log is a compacted checkpoint.

Backups and PITR restores write a binary WAL; restoring into a build that predates the binary WAL (v1.5 and earlier) needs the WAL rollback procedure first. For the full backup/incremental/PITR model, see Backup & Restore.

6. Encryption key rotation

The only way to change the key is to move the data through a client:

1. Start a fresh node (empty data dirs) with --encrypt-at-rest and the new key.
2. Read every key from a node still running with the old key and write it into the new node (for example veltrix-migrate export/import over the binary protocol).
3. Cut clients over to the new node, then retire the old one and its key.

On v1.13 and earlier, values written by atomic ops (CAS/INCR/DECR/SETNX) are stored unencrypted regardless of the key (known gap). Fixed in v1.14 they are encrypted like PUT; values an older build wrote stay plaintext until their next update.

7. Failed disk — auto-degraded node (SEV-1)

VeltrixDB trips a per-disk breaker after 5 consecutive I/O errors on a disk's WAL/VLog. The disk is marked FAILED: writes routed to it fail fast with ErrDiskFailed, GC skips it, and the node reports degraded. Reads still try the device and are served from cache when possible. Detection:

# Readiness probe fails while the node keeps serving reads
kubectl port-forward -n veltrixdb POD 2112:2112 &
curl -s -w '%{http_code}\n' localhost:2112/readyz
# degraded: disks [2] failed
# 503

# Which disks tripped
kubectl logs -n veltrixdb POD | grep '\[disk\]'   # "disk=2 marked FAILED after 5 consecutive ... errors"

INFO (text protocol) also appends FAILED_DISKS=[2] for a degraded node, and the engine's internal DiskFailures counter records breaker trips (not exported to Prometheus).

Recovery sequence:

1. Drain: the failing /readyz already stops new traffic; let in-flight work complete, then take the node out of rotation (cordon the K8s node or scale traffic away).
2. Replace: swap the failed device (see section 1's bad magic row and section 2 for corruption handling on the same disk).
3. Restart: restart the pod/process — the breaker state is in-memory, so a restart with a healthy disk comes back clean. In raft mode the leader refills the replaced disk's data through log catch-up or InstallSnapshot; in standalone mode restore from backup; in replicated mode there is no full-state resync, so restore or re-copy the data.

8. Value-transform metadata damage from a pre-fix build (SEV-1) Fixed in v1.3

Affects nodes that ran a build predating the value-transform WAL fix with compression (zstd is the default) or -encrypt-at-rest, and that restarted since writing. After a clean restart, reads succeed but silently return the raw compressed or encrypted blob (a 1040-byte value comes back as ~51 bytes starting 02 28 b5 2f fd, the zstd magic); after a crash, reads fail with vlog CRC32C mismatch. Upgrading alone does not fix it; the VLog bytes are intact, so it is repaired in place:

# 1. Assess, read-only (whole fleet; exits non-zero if any node needs work)
./scripts/repair-scan-fleet.sh --hosts hosts.txt --data-dirs /mnt/nvme0,/mnt/nvme1

# 2. Repair each node with the server STOPPED
kubectl scale statefulset veltrixdb --replicas=0
veltrix-repair --data-dirs /mnt/nvme0,/mnt/nvme1 --repair
# encrypted: add --encrypt-at-rest (key from VELTRIXDB_ENCRYPTION_KEY or --encryption-key-path)
# data not written with zstd: --compression=none|flate

Each wal.log is copied to wal.log.prerepair.<timestamp> first. The repair is exact: it accepts only the interpretation of each blob that reproduces the stored plaintext CRC, and is idempotent. Verify by reading back values larger than 256 bytes and re-running the scan. Encrypted deployments need the key (VELTRIXDB_ENCRYPTION_KEY plus --encrypt-at-rest), or those records report as unresolved. See veltrix-repair.

9. WAL format: upgrade and rollback New in v1.6

WAL records are binary by default. Replay reads binary and legacy text records in any mix, so upgrading needs no step: the new build replays the existing text WAL and appends binary records after it. A build that predates the binary WAL cannot read it. To roll back:

1. On the current build, restart once with --wal-format=text (add it to the StatefulSet args and re-roll). The server logs [wal] WARNING: --wal-format=text ….
2. Stop it cleanly (SIGTERM, not SIGKILL): kubectl scale statefulset/veltrixdb --replicas=0 -n veltrixdb. The shutdown checkpoint writes a compacted wal.log.ckpt and renames it over wal.log, now in text.
3. Deploy the older build.

If the node crashed instead of stopping cleanly, repeat steps 1–2. While running with --wal-format=text, keys containing | or a newline are not crash-safe: a crash can drop every acknowledged write after such a key.

Fixed in v1.14 the WAL record of a key with a TTL carries its expiry (binary record version 2, or an 11th text field), which no older build can read — it stops replay there and drops every later record. To roll back to v1.13 or older, use --wal-format=text-legacy in step 1 instead of text (which keeps the TTL field and is not a rollback mode). TTL’d keys come back without an expiry on the older build. On v1.13 and earlier the steps above are unchanged. Rolling back to v1.14 or later, v1.16 included, needs no WAL step: those builds read binary version-1 and version-2 records and keep TTLs, after a clean stop or a crash. Do not run text-legacy for them — it only makes every TTL’d key immortal.

Fixed in v1.15 Raft state moves from raft_state.gob to raft_log.dat + raft_meta.dat; the old file is migrated on first start, and v1.14 and earlier cannot read the new files. To roll a raft node back, stop it, delete its whole --data directory, start the older build with the same --node-id / --peers and let it re-sync from the leader, one node at a time. Do not start an older binary on a migrated directory. See Raft log persistence.

Fixed on main after v1.16 — ships in the next release the raft snapshot moves from raft_snapshot.gob to raft_snapshot.dat (migrated on start). Rolling a raft node of that release back to v1.15 or v1.16: a node with no raft_snapshot.dat starts as it is; a node with one must be wiped and re-synced as above (v1.16 refuses it with corrupt raft log … does not follow the snapshot (index 0)). Roll back followers first and the leader last. For a rolling upgrade, see Deployment Modes.

10. Raft quorum lost for good New on main after v1.16 — ships in the next release

v1.16 and earlier have no --force-new-cluster flag. On main, if a quorum is permanently gone (e.g. two of three voters destroyed): stop every survivor and start one of them with --force-new-cluster. It appends a configuration holding only itself in a new term, elects itself and commits its whole log — including entries that were never committed. Remove the flag once it is up, then wipe every other former member and add it back with --join (see the membership runbook). Never force two nodes, and never force while the old quorum may still be running. In replicated mode --force-new-cluster moves cluster_membership.json aside and starts from --peers; do it on every node with the same --peers.

Contact

• On-call: @veltrixdb-oncall in PagerDuty
• Slack: #veltrixdb-incidents
• Runbook owner: SRE team — review quarterly