Kubernetes Operator

The VeltrixDB Kubernetes Operator manages VeltrixCluster custom resources — provisioning the StatefulSet, NVMe storage, and networking, and reconciling scale, upgrade, and recovery events without manual intervention.

Overview

The Operator is built with Kubebuilder and watches a single CRD, VeltrixCluster (group veltrix.io/v1alpha1, short name vc). On each reconcile pass it creates or updates the child resources needed to run a cluster: a StorageClass, an NVMe-provisioning DaemonSet, a ServiceAccount, headless and client Service objects, a ConfigMap, a ServiceMonitor (best-effort, if the Prometheus Operator CRDs are installed), and the StatefulSet itself.

Prerequisites

Installation

From a checkout of the Operator repository, install the CRDs, then deploy the operator's controller manager:

git clone https://github.com/VeltrixDB/VeltrixDB-Kubernetes-Operator.git
cd VeltrixDB-Kubernetes-Operator

# 1. Install the CustomResourceDefinitions
kubectl apply -f config/crd/bases/

# 2. Deploy the operator (controller manager Deployment + RBAC)
kubectl apply -f config/manager/manager.yaml

Before creating any VeltrixCluster, deploy the NVMe mount-keeper DaemonSet so real XFS filesystems are bind-mounted at the PV source paths before kubelet bind-mounts them into the veltrixdb pod. Skipping this step causes ENOSPC on the very first WAL write.

kubectl apply -f config/manager/nvme-mount-keeper.yaml
kubectl rollout status daemonset/nvme-mount-keeper -n veltrixdb

See NVMe Disk Provisioning for why the keeper is required on GKE COS, AWS Bottlerocket, and any OS where kubelet runs inside its own mount namespace.

VeltrixCluster spec reference

The fields below were read from the VeltrixClusterSpec Go type (api/v1alpha1/veltrixcluster_types.go, Operator repo; not re-checked for this release). Defaults marked operator default are filled in by the reconcile loop's applyDefaults step when the field is left unset in the CR.

Core

Networking

Scheduling

Storage engine tuning

WAL / VLog flush

Security

Raw NVMe block-device VLog

Set rawVLog to write the VLog directly to /dev/nvmeXnY block devices, skipping the filesystem for the value-bytes path. WAL and segment files still use the per-pod XFS PVCs.

Example CustomResource

Adapted from the operator's shipped production sample (config/samples/veltrix_v1alpha1_veltrixcluster.yaml) — 3 nodes, 480 GiB RAM, 8 NVMe disks each, replication factor 3, raw block-device VLog with a 20/80 XFS/raw partition split:

apiVersion: veltrix.io/v1alpha1
kind: VeltrixCluster
metadata:
  name: veltrixdb-prod
  namespace: veltrixdb
spec:
  # Cluster topology
  size: 3
  version: "1.0.0"
  image: "ghcr.io/veltrixdb/veltrixdb"
  replicationFactor: 3

  # Cache — 256 GiB LIRS cache per node (80% rule for 480 GiB nodes)
  cacheMaxSizeMB: 262144

  # Storage
  storageClassName: local-nvme
  disksPerNode: 8
  diskSize: "375Gi"

  # Resources (n2-highmem-96 class)
  resources:
    requests:
      memory: "480Gi"
      cpu: "60"
    limits:
      memory: "480Gi"
      cpu: "60"

  # Network ports
  serverPort: 9000
  metricsPort: 2112
  gossipPort: 7946

  # Raw NVMe block-device VLog: 20% XFS (keys+WAL), 80% raw (values)
  rawVLog:
    enabled: true
    xfsPartitionPercent: 20

  # Scheduling — one pod per physical host
  podAntiAffinityTopologyKey: "kubernetes.io/hostname"
  nodeSelector:
    cloud.google.com/gke-local-nvme-ssd: "true"
  tolerations:
    - key: workload
      value: veltrixdb
      effect: NoSchedule

kubectl apply -f veltrixcluster.yaml
kubectl get veltrixcluster -n veltrixdb

Status fields

kubectl describe veltrixcluster my-cluster -n veltrixdb

What the reconcile loop actually does

This section is scoped to what is verifiably implemented in controllers/veltrixcluster_controller.go, not just described in prose elsewhere.

Child resource management

Each reconcile pass creates or updates, in order: the local-nvme StorageClass, an NVMe-provisioner DaemonSet (plus its ServiceAccount/ClusterRole/ClusterRoleBinding), the pod ServiceAccount, headless and client Service objects, a best-effort ServiceMonitor (a reconcile failure here only logs a warning — it does not fail the whole reconcile, since the Prometheus Operator CRDs may not be installed), the ConfigMap of engine env vars, and finally the StatefulSet.

Rolling upgrades — confirmed

The StatefulSet is built with updateStrategy.type: RollingUpdate and rollingUpdate.maxUnavailable: 1. Bumping spec.version (or spec.image) and re-applying the CR causes Kubernetes' native StatefulSet controller to replace one pod at a time, waiting for each to become ready before proceeding — the operator does not implement custom upgrade orchestration beyond setting this strategy.

kubectl patch veltrixcluster my-cluster -n veltrixdb \
  --type merge --patch '{"spec":{"version":"1.1.0"}}'

Auto-reshard on replica count changes — confirmed, with caveats

The reconciler compares the StatefulSet's previous replica count to spec.size on every pass. When the cluster has grown (prevSize < spec.size), it calls triggerResharding, which:

1. Blocks (up to 5 minutes) until all desired pods report PodReady.
2. Annotates every pod with veltrix.io/cluster-size and veltrix.io/prev-cluster-size. Server fact: nothing in the VeltrixDB server reads these annotations. A server builds its membership once at startup from --peers (or --seeds), gossip does not add members, and there is no runtime join command; on v1.14 and earlier --auto-rebalance migrates keys only when that static membership changes state (a node fails or recovers). Fixed in v1.15 state changes never migrate and raft mode never migrates, so with static membership no keys move. In practice, on v1.16 and earlier a new node joins the ring only when every pod restarts with it listed in --peers. New on main after v1.16 — ships in the next release the server can add and remove members online (/admin/cluster/members, --join, --advertise-addr), but the operator does not use it yet: it still scales by restarting every pod with a new --peers, which stops working once a change was made online (persisted membership wins over --peers). The server repo’s intended pattern: keep --peers at the bootstrap members, add --join=<bootstrap pod addresses> and --advertise-addr=$(POD_FQDN):<port> to every pod, scale up by raising the replica count, and scale down by calling DELETE /admin/cluster/members/<pod> for the highest ordinal before lowering the count. See the membership runbook.
3. Records status.lastScaleTime and a status.partitionMap summary string.

Self-healing — partially confirmed

The operator itself does not run a custom pod-replacement loop; it relies on the standard StatefulSet controller to recreate any pod that is deleted or fails its liveness probe, which is the same "self-healing" every StatefulSet gets from Kubernetes. What the operator adds on top:

• It stamps every pod with cluster-autoscaler.kubernetes.io/safe-to-evict: "false", specifically to stop the Cluster Autoscaler from deleting nodes that carry irreplaceable local NVMe PVs.
• Its periodic requeue (every 30 seconds) re-evaluates readyReplicas and updates the Ready/ReplicationFactorMet conditions, surfacing drift for operators/alerting rather than performing autonomous data-rebuild actions itself.

Graceful WAL flush on termination — confirmed

Every pod gets a preStop lifecycle hook (kill -SIGTERM 1 && sleep 55 — note the official image is distroless with no shell, so an exec hook of this form cannot run there; the kubelet's own SIGTERM still triggers the server's clean shutdown) and a 60-second terminationGracePeriodSeconds, giving the engine time to flush its WAL before the container is killed. On VeltrixCluster deletion, the operator's finalizer (veltrix.io/finalizer) annotates pods as draining before removing itself — the actual flush is still performed by the pod's own SIGTERM handling and the grace period, not a separate operator-issued flush command.

Scaling

kubectl patch veltrixcluster my-cluster -n veltrixdb \
  --type merge --patch '{"spec":{"size":5}}'

Increasing spec.size triggers the StatefulSet to add pods and, once they are ready, the annotation step described above. Whether the existing pods then learn about the new ones depends on the --peers list your Operator version renders and on a restart; the server does not discover peers at runtime.

Node, seccomp and image notes

• Server fact: a clean SIGTERM shutdown writes a compacted WAL checkpoint; budget the grace period for it plus, on the next start, the search-index rebuild (searches are refused until INFO shows search_ready=1; /readyz does not wait for it).
• The official image is built with CGO_ENABLED=1 (distroless cc-debian12 base), so the server uses the off-heap native index by default. New in v1.6 It needs vm.max_map_count ≥ 262144 on the node at large key counts — a container cannot set it. Configure it in the node pool / node image or a privileged DaemonSet, or set VELTRIXDB_INDEX=map in the pod env.
• The RuntimeDefault seccomp profile blocks io_uring_setup. That only matters for the opt-in io_uring VLog write bridge (VELTRIXDB_URING_BRIDGE=on|sqpoll; default off), which then falls back to pwrite. The rawVLog.enabled path above already sets seccompProfile: Unconfined.
• The experimental C++ network front-end (-net=cpp|uring|poll) is refused in -mode raft and -mode replicated and with -auth-config, so any multi-node or RBAC-enabled cluster must keep the default -net=go.

Troubleshooting

Related pages