Kubernetes Operator

The VeltrixDB Kubernetes Operator manages VeltrixCluster custom resources — provisioning the StatefulSet, NVMe storage, and networking, and reconciling scale, upgrade, and recovery events without manual intervention.

Overview

The Operator is built with Kubebuilder and watches a single CRD, VeltrixCluster (group veltrix.io/v1alpha1, short name vc). On each reconcile pass it creates or updates the child resources needed to run a cluster: a StorageClass, an NVMe-provisioning DaemonSet, a ServiceAccount, headless and client Service objects, a ConfigMap, a ServiceMonitor (best-effort, if the Prometheus Operator CRDs are installed), and the StatefulSet itself.

Prerequisites

Installation

Install the CRDs, then deploy the operator's controller manager:

# 1. Install the CustomResourceDefinitions
kubectl apply -f config/crd/bases/

# 2. Deploy the operator (controller manager Deployment + RBAC)
kubectl apply -f config/manager/manager.yaml

Before creating any VeltrixCluster, deploy the NVMe mount-keeper DaemonSet so real XFS filesystems are bind-mounted at the PV source paths before kubelet bind-mounts them into the veltrixdb pod. Skipping this step causes ENOSPC on the very first WAL write.

kubectl apply -f config/manager/nvme-mount-keeper.yaml
kubectl rollout status daemonset/nvme-mount-keeper -n veltrixdb

See NVMe Disk Provisioning for why the keeper is required on GKE COS, AWS Bottlerocket, and any OS where kubelet runs inside its own mount namespace.

VeltrixCluster spec reference

The fields below are read directly from the VeltrixClusterSpec Go type (api/v1alpha1/veltrixcluster_types.go). Defaults marked operator default are filled in by the reconcile loop's applyDefaults step when the field is left unset in the CR.

Core

Networking

Scheduling

Storage engine tuning

WAL / VLog flush

Security

Raw NVMe block-device VLog

Set rawVLog to write the VLog directly to /dev/nvmeXnY block devices, skipping the filesystem for the value-bytes path. WAL and segment files still use the per-pod XFS PVCs.

Example CustomResource

Adapted from the operator's shipped production sample (config/samples/veltrix_v1alpha1_veltrixcluster.yaml) — 3 nodes, 480 GiB RAM, 8 NVMe disks each, replication factor 3, raw block-device VLog with a 20/80 XFS/raw partition split:

apiVersion: veltrix.io/v1alpha1
kind: VeltrixCluster
metadata:
  name: veltrixdb-prod
  namespace: veltrixdb
spec:
  # Cluster topology
  size: 3
  version: "1.0.0"
  image: "ghcr.io/veltrixdb/veltrixdb"
  replicationFactor: 3

  # Cache — 256 GiB LIRS cache per node (80% rule for 480 GiB nodes)
  cacheMaxSizeMB: 262144

  # Storage
  storageClassName: local-nvme
  disksPerNode: 8
  diskSize: "375Gi"

  # Resources (n2-highmem-96 class)
  resources:
    requests:
      memory: "480Gi"
      cpu: "60"
    limits:
      memory: "480Gi"
      cpu: "60"

  # Network ports
  serverPort: 9000
  metricsPort: 2112
  gossipPort: 7946

  # Raw NVMe block-device VLog: 20% XFS (keys+WAL), 80% raw (values)
  rawVLog:
    enabled: true
    xfsPartitionPercent: 20

  # Scheduling — one pod per physical host
  podAntiAffinityTopologyKey: "kubernetes.io/hostname"
  nodeSelector:
    cloud.google.com/gke-local-nvme-ssd: "true"
  tolerations:
    - key: workload
      value: veltrixdb
      effect: NoSchedule

kubectl apply -f veltrixcluster.yaml
kubectl get veltrixcluster -n veltrixdb

Status fields

kubectl describe veltrixcluster my-cluster -n veltrixdb

What the reconcile loop actually does

This section is scoped to what is verifiably implemented in controllers/veltrixcluster_controller.go, not just described in prose elsewhere.

Child resource management

Each reconcile pass creates or updates, in order: the local-nvme StorageClass, an NVMe-provisioner DaemonSet (plus its ServiceAccount/ClusterRole/ClusterRoleBinding), the pod ServiceAccount, headless and client Service objects, a best-effort ServiceMonitor (a reconcile failure here only logs a warning — it does not fail the whole reconcile, since the Prometheus Operator CRDs may not be installed), the ConfigMap of engine env vars, and finally the StatefulSet.

Rolling upgrades — confirmed

The StatefulSet is built with updateStrategy.type: RollingUpdate and rollingUpdate.maxUnavailable: 1. Bumping spec.version (or spec.image) and re-applying the CR causes Kubernetes' native StatefulSet controller to replace one pod at a time, waiting for each to become ready before proceeding — the operator does not implement custom upgrade orchestration beyond setting this strategy.

kubectl patch veltrixcluster my-cluster -n veltrixdb \
  --type merge --patch '{"spec":{"version":"1.1.0"}}'

Auto-reshard on replica count changes — confirmed, with caveats

The reconciler compares the StatefulSet's previous replica count to spec.size on every pass. When the cluster has grown (prevSize < spec.size), it calls triggerResharding, which:

1. Blocks (up to 5 minutes) until all desired pods report PodReady.
2. Annotates every pod with veltrix.io/cluster-size and veltrix.io/prev-cluster-size. The VeltrixDB binary is expected to watch these annotations via the downward API and recompute its consistent-hash partition map / trigger the actual shard rebalance — the operator's role is limited to triggering the reshard, not performing the data movement itself.
3. Records status.lastScaleTime and a status.partitionMap summary string.

Self-healing — partially confirmed

The operator itself does not run a custom pod-replacement loop; it relies on the standard StatefulSet controller to recreate any pod that is deleted or fails its liveness probe, which is the same "self-healing" every StatefulSet gets from Kubernetes. What the operator adds on top:

• It stamps every pod with cluster-autoscaler.kubernetes.io/safe-to-evict: "false", specifically to stop the Cluster Autoscaler from deleting nodes that carry irreplaceable local NVMe PVs.
• Its periodic requeue (every 30 seconds) re-evaluates readyReplicas and updates the Ready/ReplicationFactorMet conditions, surfacing drift for operators/alerting rather than performing autonomous data-rebuild actions itself.

Graceful WAL flush on termination — confirmed

Every pod gets a preStop lifecycle hook (kill -SIGTERM 1 && sleep 55) and a 60-second terminationGracePeriodSeconds, giving the engine time to flush its WAL before the container is killed. On VeltrixCluster deletion, the operator's finalizer (veltrix.io/finalizer) annotates pods as draining before removing itself — the actual flush is still performed by the pod's own SIGTERM handling and the grace period, not a separate operator-issued flush command.

Scaling

kubectl patch veltrixcluster my-cluster -n veltrixdb \
  --type merge --patch '{"spec":{"size":5}}'

Increasing spec.size triggers the StatefulSet to add pods and, once they are ready, the reshard-trigger flow described above.

Troubleshooting

Related pages