Disaster Recovery
A runbook-style summary of VeltrixDB's disaster recovery procedures, covering pod crash loops, data corruption, write outages, GC death-spirals, backup/restore, and encryption key rotation. Assumes kubectl admin access with pods labeled app.kubernetes.io/name=veltrixdb in the veltrixdb namespace.
Severity classes
| Class | Definition | Engage on-call |
|---|---|---|
| SEV-0 | Cluster-wide write outage or confirmed data loss | < 5 min |
| SEV-1 | One pod down, replicas serving | < 15 min |
| SEV-2 | Read latency > 5x baseline | < 1 hour |
| SEV-3 | Single disk slow, scrubber alerting, no user impact | < 4 hours |
1. Pod crash loop (SEV-1)
Start by pulling logs and pod status:
kubectl logs -n veltrixdb POD --previous | tail -100
kubectl describe pod -n veltrixdb POD| Log line | Cause | Fix |
|---|---|---|
WAL replay corruption at offset N | Partial write on crash | Restart the pod once; if recurring, run kubectl veltrix sync-from REPLICA_POD |
cannot allocate memory (vlog open) | Hugepages missing | kubectl exec POD -- sysctl vm.nr_hugepages — must be ≥ 512 |
bad magic at offset O | Silent disk corruption | Drain the node, replace the disk, re-join |
encryption: no key in VELTRIXDB_ENCRYPTION_KEY | Missing secret | kubectl create secret generic veltrixdb-enc --from-literal=key=BASE64_32B |
2. Data corruption (SEV-0)
- Confirm corruption via metrics:
kubectl exec -n veltrixdb POD -- curl -s localhost:2112/metrics | \ grep -E 'scrub_corruption_total|vlog_gc_read_errors_total' - Find the bad disk:
kubectl logs -n veltrixdb POD | grep '[scrub] disk=' - Quarantine the pod:
kubectl label pod -n veltrixdb POD veltrixdb.io/quarantine=true --overwrite - Wipe the affected files and restart — replication refills automatically:
kubectl exec -n veltrixdb POD -- rm -rf /mnt/nvme*/vlog_active.dat /mnt/nvme*/wal_* kubectl delete pod -n veltrixdb POD - After replay completes, remove the quarantine label:
kubectl label pod -n veltrixdb POD veltrixdb.io/quarantine- --overwrite - Root-cause the disk:
kubectl exec -n veltrixdb POD -- smartctl -a /dev/nvme0n1
3. Write outage (SEV-0)
kubectl get pods -n veltrixdb -o wide
kubectl exec -n veltrixdb POD -- df -h /mnt/nvme*
kubectl exec -n veltrixdb POD -- curl -s localhost:2112/metrics | grep wal_flushes_total| Cause | Fix |
|---|---|
| Disk 100% full and GC paused | kubectl veltrix checkpoint |
| GC can't keep up | kubectl veltrix quota-set NS 1000 5000000 — throttle writes |
| Network partition | kubectl exec POD -- nc -zv OTHER_POD 9000 — check CNI/NetworkPolicies |
4. GC death-spiral / garbage ratio > 65% (SEV-2)
- Throttle writes immediately:
kubectl veltrix quota-set tenant_42 1000 5000000 - If hardware has headroom, raise the GC budget:
kubectl set env statefulset/veltrixdb -n veltrixdb \ VELTRIXDB_GC_CRITICAL_BPS=300000000 # 300 MB/s - Long-term: add disks (scale
volumeClaimTemplates, re-roll the StatefulSet).
Emergency-mode GC keeps the system upright automatically once the garbage ratio crosses this threshold — the operator's job during this incident is to reduce write rate before the disk fills, not to force GC harder.
5. Backup and restore
# Full backup (engine can be running)
veltrixdb-backup full --data-dirs=/mnt/nvme0,...,/mnt/nvme7 --dest=/backup/$(date +%F)
# Upload to S3
veltrixdb-backup upload --src=/backup/$(date +%F) \
--provider=s3 --bucket=my-bucket --region=us-east-1
# Download and restore (stop the engine first)
veltrixdb-backup download --provider=s3 --bucket=my-bucket \
--cloud-path=veltrixdb-backups/$(date +%F) --dest=/tmp/restore
veltrixdb-backup restore --chain=/tmp/restore --data-dirs=/data-newOr restore from Kubernetes volume snapshots:
kubectl create volumesnapshot veltrixdb-disk-0 -n veltrixdb \
--persistentvolumeclaim=data-veltrixdb-0For the full backup/incremental/PITR model, see Backup & Restore.
6. Encryption key rotation
Online rotation is not supported. Rotating the AES-256-GCM master key requires taking the cluster offline for the duration of the migration.
- Stop all pods:
kubectl scale statefulset/veltrixdb --replicas=0 -n veltrixdb - Generate a new key:
openssl rand 32 | base64 - Update the secret:
kubectl create secret generic veltrixdb-enc --from-literal=key=NEW_KEY --dry-run=client -o yaml | kubectl apply -f - - Start the pods and force a full rewrite at the new key:
kubectl veltrix migrate
7. Failed disk — auto-degraded node (SEV-1)
VeltrixDB trips a per-disk breaker after 5 consecutive I/O errors on a disk's WAL/VLog. The disk is marked FAILED: writes routed to it fail fast with ErrDiskFailed, GC skips it, and the node reports degraded. Reads still try the device and are served from cache when possible. Detection:
# Readiness probe fails while the node keeps serving reads
kubectl exec -n veltrixdb POD -- curl -s -o /dev/null -w '%{http_code}\n' localhost:2112/readyz
# 503 — body: "degraded: disks [2] failed"
# Which disks tripped
kubectl exec -n veltrixdb POD -- curl -s localhost:2112/readyz # degraded: disks [2] failed
kubectl logs -n veltrixdb POD | grep '\[disk\]' # "disk=2 marked FAILED after 5 consecutive ... errors"INFO (text protocol) also appends FAILED_DISKS=[2] for a degraded node, and the engine's DiskFailures metric counts breaker trips.
The breaker does not auto-reset. A failed disk is never automatically un-failed — even if later I/O on it would succeed. Recovery is always drain / replace / restart, which matches how hardware is actually swapped.
Recovery sequence:
- Drain: the failing
/readyzalready stops new traffic; let in-flight work complete, then take the node out of rotation (cordon the K8s node or scale traffic away). - Replace: swap the failed device (see section 1's
bad magicrow and section 2 for corruption handling on the same disk). - Restart: restart the pod/process — the breaker state is in-memory, so a restart with a healthy disk comes back clean and replication refills the data.
Contact
- On-call:
@veltrixdb-oncallin PagerDuty - Slack:
#veltrixdb-incidents - Runbook owner: SRE team — review quarterly