Helm Chart

The VeltrixDB Helm chart deploys a production-ready cluster on Kubernetes with per-pod local NVMe storage, Prometheus monitoring, and safe rolling updates — optimized for GKE but portable to any cloud with local NVMe instance types.

Prerequisites

gcloud container node-pools create veltrixdb-pool \
  --cluster=my-cluster \
  --zone=us-central1-a \
  --machine-type=n2-highmem-64 \
  --num-nodes=3 \
  --local-ssd-count=8 \
  --local-ssd-interface=NVME \
  --disk-size=100GB

Installation

# Add the chart repo
helm repo add veltrixdb https://charts.veltrixdb.io
helm repo update

# Install with defaults
helm install veltrixdb veltrixdb/veltrixdb \
  --namespace veltrixdb \
  --create-namespace

# Install with custom values
helm install veltrixdb veltrixdb/veltrixdb \
  --namespace veltrixdb \
  --create-namespace \
  --values my-values.yaml

Verify the deployment

# Wait for PVs (3 nodes x 8 disks = 24 volumes)
kubectl get pv | grep local-nvme

# Check pods are running
kubectl get pods -n veltrixdb -l app.kubernetes.io/name=veltrixdb

# Test connectivity (the image is distroless: no shell or nc inside the pod)
kubectl port-forward -n veltrixdb veltrixdb-0 9000:9000 &
echo PING | nc 127.0.0.1 9000
# Expected: PONG

What the chart deploys

Per the templates in VeltrixDB-Helm-Chart/templates/ (chart repo): statefulset.yaml, service.yaml, service-headless.yaml, servicemonitor.yaml, pdb.yaml, prometheusrule.yaml, configmap.yaml, secret.yaml, rbac.yaml, and _helpers.tpl.

• StatefulSet — VeltrixDB pods with stable per-pod hostnames (podManagementPolicy: Parallel, one volume-claim template per NVMe disk).
• Headless + client Services — the headless service backs gossip/StatefulSet DNS (publishNotReadyAddresses: true so bootstrap peers are reachable pre-readiness); the client service fronts application traffic.
• ServiceMonitor — Prometheus Operator scrape config, one target per pod.
• PodDisruptionBudget — bounds voluntary disruptions during node drains and upgrades.
• PrometheusRule — a bundled Prometheus alerting ruleset (see below).
• ConfigMap / Secret — engine configuration env vars and a bootstrap token. Server fact: the server reads no bootstrap token. The node-to-node cluster secret it does read (VELTRIXDB_CLUSTER_SECRET or --cluster-secret-file, ≥ 16 bytes, new in v1.11) must be wired into the pod yourself if your chart version has no value for it; without it (or cluster mTLS) the transfer listener is unauthenticated.
• RBAC — ServiceAccount plus any Role/RoleBinding the pods need.

Pod Disruption Budget

The chart's default values.yaml sets podDisruptionBudget.maxUnavailable: 1 — at most one pod may be voluntarily disrupted at a time, which for a 3-node, RF-3 cluster keeps at least 2 nodes available during drains and rolling upgrades.

podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

PrometheusRule alerts

The chart's PrometheusRule (rendered when serviceMonitor.enabled: true; chart repo, not re-verified) was counted at 24 alert rules in 10 rule groups, spanning write/read/delete latency, VLog GC health, cluster membership, replication, per-disk health, background scrubbing, and Bloom filter efficiency.

The most operationally important two: VeltrixCorruptionDetected fires the moment the background scrubber finds a CRC32C or magic mismatch (severity critical, for: 0m) — treat as an immediate SMART-check-and-resync trigger. VeltrixGCEmergencyRunsSustained fires when VLog garbage has sat above the emergency threshold (65%) for 30 minutes and GC is bypassing admission control just to keep up — it means sustained write rate is exceeding GC throughput and you need to reduce load or add disks.

Configuration reference

Key settings from values.yaml. Full defaults are tuned for GKE n2-highmem-96 nodes (480 GB RAM, 96 vCPU, 8 x 375 GiB local NVMe SSDs).

Performance tuning

For 100K+ writes/s, lower the flush windows (both must stay equal):

storage:
  walFlushWindowMs: 5
  vlogFlushWindowMs: 5

For maximum cache hit rate, give the cache most of the node's RAM, leaving headroom for the index (~142 B/key with the native index), the Go heap and the OS (the server's -auto-tune uses 85% of 80% of RAM):

cache:
  maxSizeMB: 409600  # 400 GB on a 512 GB node

Node, seccomp and image notes

• Image. The official image is built with CGO_ENABLED=1 on a gcr.io/distroless/cc-debian12 base, so it uses the off-heap native index by default. New in v1.6 Build with --build-arg CGO_ENABLED=0 for the old static pure-Go image (Go map index). cgo code is compiled for -march=x86-64-v2 on amd64, never -march=native.
• vm.max_map_count must be set on the node. The native index needs vm.max_map_count ≥ 262144 at large key counts. It is a node-level sysctl that a container cannot set, so configure it in the node pool / node image or with a privileged DaemonSet (scripts/sysctl.conf has the value). Otherwise set VELTRIXDB_INDEX=map in the pod env.
• seccomp and io_uring. The Kubernetes RuntimeDefault seccomp profile blocks io_uring_setup. That only matters if you opt in to the io_uring VLog write bridge (VELTRIXDB_URING_BRIDGE=on|sqpoll; default off), and even then the bridge falls back to pwrite. To actually use it, the pod needs a seccomp profile allowing io_uring_setup, io_uring_enter and io_uring_register. The default configuration and the native index need nothing special.

Security, audit, and data-integrity features

TLS and RBAC auth

auth:
  enabled: true
  configSecret: veltrixdb-auth

tls:
  enabled: true
  existingSecret: veltrixdb-tls

At-rest encryption (AES-256-GCM)

kubectl create secret generic veltrixdb-encryption \
  -n veltrixdb \
  --from-literal=key=$(openssl rand 32 | base64)

security:
  encryptionAtRest: true
  encryptionSecretName: veltrixdb-encryption

Each pod loads the key from the VELTRIXDB_ENCRYPTION_KEY env var at startup; without the secret, startup fails when security.encryptionAtRest is true. Key rotation is not implemented — never change the Secret under existing data (see Disaster Recovery). On v1.13 and earlier, atomic ops (CAS/INCR/DECR/SETNX) store their values unencrypted. Fixed in v1.14 they are encrypted like PUT.

Background scrubber

scrubber:
  enabled: true
  mbPerSec: 50

CRC32C re-validation of every VLog record at a controlled rate (server -scrub-mb-per-sec, default 50 MB/s per disk; 0 disables it); 50 MB/s is roughly a 2-hour full pass over 375 GB. Mismatches increment veltrixdb_scrub_corruption_total and log [scrub] disk=N offset=M … MISMATCH.

Raw NVMe block-device VLog

For latency-critical clusters, enable storage.rawVlog.enabled to write the VLog directly to block devices instead of XFS files. WAL and segment files continue to live on the XFS PVCs.

storage:
  rawVlog:
    enabled: true
    devices:
      - /dev/nvme0n1
      - /dev/nvme1n1
      - /dev/nvme2n1
      - /dev/nvme3n1
      - /dev/nvme4n1
      - /dev/nvme5n1
      - /dev/nvme6n1
      - /dev/nvme7n1

Server requirements for this mode (root, CAP_SYS_RAWIO, one device per data dir, the 4 KB superblock) are on NVMe Disk Provisioning. No raw-vs-XFS latency comparison is published in the main repository, and online backups contain no VLog data in this mode.

Production values.yaml example

replicaCount: 3

image:
  tag: "1.12"        # set to the server release you run; check the tag format your registry uses

storage:             # one storage: block (a second one would replace the first)
  disksPerNode: 8
  diskSizeGi: 375
  walFlushWindowMs: 10
  vlogFlushWindowMs: 10

cache:
  maxSizeMB: 262144  # 256 GiB - recommended for 480 GiB nodes

resources:
  requests:
    memory: 480Gi
    cpu: "60"
  limits:
    hugepages-2Mi: 2Gi

autoTune:
  enabled: false

cluster:
  gossipPort: 7946
  replicationFactor: 3

auth:
  enabled: true
  configSecret: veltrixdb-auth

tls:
  existingSecret: veltrixdb-tls

serviceMonitor:
  enabled: true
  interval: 15s

podDisruptionBudget:
  maxUnavailable: 1

Upgrading

helm upgrade veltrixdb veltrixdb/veltrixdb \
  --namespace veltrixdb \
  --values my-values.yaml

The StatefulSet uses RollingUpdate with maxUnavailable: 1, so pods are replaced one at a time. The PodDisruptionBudget bounds voluntary disruptions from node drains that happen concurrently with the rollout. Server facts for upgrades: the binary WAL needs no upgrade step (rolling back to v1.5 or earlier needs one clean restart with --wal-format=text; Fixed in v1.14 rolling back from that release to v1.13 or older needs --wal-format=text-legacy instead, because TTL’d keys’ WAL records carry their expiry; v1.14 and later, v1.16 included, need no WAL step). In raft mode upgrade followers first and the leader last, one pod at a time, each only after the restarted pod has applied a fresh write — maxUnavailable: 1 does not wait for that (see Rolling upgrade), and after each restart searches are refused until the node's index rebuild finishes (INFO → search_ready=1), which /readyz does not wait for.

Validating a deployment

scripts/bench.sh in the main repository is a local harness: it builds its own server and load generator, starts the server on ADDR (default 127.0.0.1:9000) and wipes /tmp data dirs, so it cannot be pointed at a deployed cluster. Run it on a node of the same hardware class to gate a build (packing density ≤ 1.2 × (24 + value size) bytes/record, zero GC emergency runs, read P99 ≤ 5 ms; exit 0 = pass, log in /tmp/veltrixbench-out/run.log).

To exercise the deployed cluster itself, drive it with the load tester through a port-forward:

kubectl port-forward -n veltrixdb veltrixdb-0 9000:9000 2112:2112 &

git clone https://github.com/VeltrixDB/veltrixdb.git && cd veltrixdb
go run ./cmd/loadtest --addr=127.0.0.1:9000 \
  --mode=mixed --concurrency=64 --duration=30 \
  --num-keys=1000000 --value-size=128 --read-ratio=0.7
curl -s localhost:2112/metrics | grep -E 'gc_emergency_runs_total|write_admission_throttles_total'

A port-forward adds its own latency; use the numbers as a smoke test, not a benchmark.

Uninstalling

helm uninstall veltrixdb --namespace veltrixdb

PVCs are not deleted automatically, to protect your data:

kubectl delete pvc -n veltrixdb -l app.kubernetes.io/name=veltrixdb

Troubleshooting

Related pages