NVMe Disk Provisioning
Why VeltrixDB needs raw access to local NVMe disks in Kubernetes, and how the mount-keeper / nvme-prep DaemonSets get real XFS filesystems in front of each pod before the engine starts.
--local-ssd-interface=NVME. Without it, GKE merges all local SSDs into a single RAID-0 device and you lose the parallel per-disk I/O that VeltrixDB's WAL/VLog layout depends on.
Why this exists
VeltrixDB's storage engine (see Storage Engine) opens one WAL and one VLog file per physical disk and expects them to sit on real, independently-addressable NVMe devices — not a filesystem abstraction shared across disks. On several managed Kubernetes platforms, the kubelet mounts local-SSD-backed PersistentVolumes into a pod's mount namespace before anything has formatted them, or formats them in a way that collapses several physical disks into one logical device. Both cases silently defeat the parallel-disk design.
The mount-keeper / nvme-prep DaemonSet exists to run on the host, format each local NVMe device as XFS (or leave it raw — see below), and bind-mount it at the exact path the PersistentVolume expects before kubelet performs its own bind-mount into the VeltrixDB pod.
Per-cloud notes
| Cloud | Requirement |
|---|---|
| GKE | Node pool created with --local-ssd-interface=NVME (see the warning above). GKE's Container-Optimized OS (COS) runs kubelet inside its own mount namespace, which is the main reason the mount-keeper pattern is needed at all. |
| AWS | Instance types with local NVMe instance storage: i3, i3en, i4i, im4gn, is4gen. Similar bind-mount timing issues occur on Bottlerocket. |
| Azure | Lsv2/Lsv3 series with local NVMe storage. |
What gets installed
Depending on how you deploy VeltrixDB, the DaemonSet is provisioned one of two ways:
- Kubernetes Operator — the Operator provisions the NVMe-provisioner DaemonSet dynamically for each
VeltrixClusteras part of its normal reconcile loop. See Kubernetes Operator. - Helm chart alone — the chart does not ship a static
daemonset.yamltemplate for this. Apply it directly beforehelm install:kubectl apply -f VeltrixDB-Kubernetes-Operator/config/manager/nvme-mount-keeper.yaml kubectl rollout status daemonset/nvme-mount-keeper -n veltrixdb
XFS-backed vs. raw block-device VLog
By default, each NVMe disk is formatted XFS and both the WAL and the VLog live as regular files on it. For latency-critical clusters, VeltrixDB can write the VLog directly to the raw block device (/dev/nvmeXnY), skipping the filesystem entirely for the value-bytes path — the WAL and segment metadata still live on an XFS partition.
| Mode | Trade-off |
|---|---|
| XFS-backed VLog (default) | Simpler operationally; periodic XFS journal commits can introduce a small P99.99 latency blip (roughly every 30s under sustained writes). |
Raw block-device VLog (rawVLog.enabled / storage.rawVlog.enabled) | Flat tail latency (no journal blips); saves roughly 25 µs off P99 read and 70 µs off P99 write. Requires the SYS_RAWIO capability, a /dev hostPath mount, and (on GKE COS) a privileged, root container to pass the cgroup device allowlist. |
mkfs.xfs on any device designated for raw VLog use. WAL/segment XFS partitions belong on a separate slice of the disk (xfsPartitionPercent, typically 20% in production) or on the boot disk.
When device paths aren't explicitly configured, both the Operator and the Helm chart infer them as /dev/nvme0n1 … /dev/nvme(disksPerNode-1)n1 — the standard GKE local-NVMe naming convention.
Troubleshooting
| Symptom | Cause / fix |
|---|---|
write /mnt/nvmeN/wal.log: no space left on device | The pod started before the mount-keeper bound real XFS at the PV source path, so kubelet bind-mounted a small tmpfs placeholder instead. Confirm the DaemonSet is Running, then delete the pod so kubelet re-creates the bind mount against the now-formatted disk. |
Pods stuck in Pending | Check that PVs are Available, the provisioner DaemonSet pods are Running, and (on GKE) that the node pool was created with --local-ssd-interface=NVME. |
| Cluster provisions but write throughput is far below benchmark numbers | Verify GKE didn't collapse local SSDs into a single RAID-0 device — this happens automatically if the node pool omits --local-ssd-interface=NVME. |