NVMe Disk Provisioning
Why VeltrixDB needs raw access to local NVMe disks in Kubernetes, and how the mount-keeper / nvme-prep DaemonSets get real XFS filesystems in front of each pod before the engine starts.
--local-ssd-interface=NVME. Without it, GKE merges all local SSDs into a single RAID-0 device and you lose the parallel per-disk I/O that VeltrixDB's WAL/VLog layout depends on.
VeltrixDB-Helm-Chart) and the Operator (VeltrixDB-Kubernetes-Operator) are separate repositories. Manifest paths, value names and CR fields on this page come from those repos and could not be checked against the server source for v1.12 — confirm them in the repo you deploy from. What the server requires (below: --raw-vlogs, device permissions, sysctls) is verified against the main repository.
Why this exists
VeltrixDB's storage engine (see Storage Engine) opens one WAL and one VLog file per physical disk and expects them to sit on real, independently-addressable NVMe devices — not a filesystem abstraction shared across disks. On several managed Kubernetes platforms, the kubelet mounts local-SSD-backed PersistentVolumes into a pod's mount namespace before anything has formatted them, or formats them in a way that collapses several physical disks into one logical device. Both cases silently defeat the parallel-disk design.
The mount-keeper / nvme-prep DaemonSet exists to run on the host, format each local NVMe device as XFS (or leave it raw — see below), and bind-mount it at the exact path the PersistentVolume expects before kubelet performs its own bind-mount into the VeltrixDB pod.
Per-cloud notes
| Cloud | Requirement |
|---|---|
| GKE | Node pool created with --local-ssd-interface=NVME (see the warning above). GKE's Container-Optimized OS (COS) runs kubelet inside its own mount namespace, which is the main reason the mount-keeper pattern is needed at all. |
| AWS | Instance types with local NVMe instance storage: i3, i3en, i4i, im4gn, is4gen. Similar bind-mount timing issues occur on Bottlerocket. |
| Azure | Lsv2/Lsv3 series with local NVMe storage. |
What gets installed
Depending on how you deploy VeltrixDB, the DaemonSet is provisioned one of two ways:
- Kubernetes Operator — the Operator provisions the NVMe-provisioner DaemonSet dynamically for each
VeltrixClusteras part of its normal reconcile loop. See Kubernetes Operator. - Helm chart alone — per the Operator repo, the chart does not ship a static
daemonset.yamltemplate for this; apply the manifest from a checkout of the Operator repository beforehelm install:kubectl apply -f VeltrixDB-Kubernetes-Operator/config/manager/nvme-mount-keeper.yaml kubectl rollout status daemonset/nvme-mount-keeper -n veltrixdb
XFS-backed vs. raw block-device VLog
By default, each NVMe disk is formatted XFS and both the WAL and the VLog live as regular files on it. The server can instead write the VLog directly to a raw block device with --raw-vlogs, one device per --data-dirs entry (paired by index; a count mismatch stops startup). The WAL and segment files stay on --data-dirs:
veltrixdb --data-dirs=/mnt/nvme0,/mnt/nvme1 \
--raw-vlogs=/dev/nvme0n1p2,/dev/nvme1n1p2What the server does with each device (Linux only):
- Opens it
O_RDWR|O_DIRECT— deliberately withoutO_EXCL, which returnsEPERMon a sibling partition of a disk whose other partition is mounted. Nothing therefore stops two processes opening the same device; the superblock check and one pod per device are the guard. - Reserves bytes [0, 4096) for a superblock (magic
0x564C4252"VLBR", version, device size, sector size, CRC32C). The first open writes it when the first 64 bytes are zero; otherwise a magic, version or CRC mismatch aborts startup rather than overwriting foreign data. Records start at offset 4096 and use the same encoding as the file VLog. - Reclaims GC'd space with the
BLKDISCARDioctl (NVMe TRIM) instead offallocate(PUNCH_HOLE), never touching the superblock. - Needs root (or equivalent access to the device node) and
CAP_SYS_RAWIOforBLKDISCARD. A missing capability does not stop the server: TRIM is silently skipped, the VLog head is never reclaimed, andveltrixdb_vlog_blkdiscard_errors_totalrises. Alert on it.
| Mode | Trade-off |
|---|---|
| XFS-backed VLog (default) | Simpler operationally; online backups (POST /admin/backup) copy vlog_active.dat and work normally. |
Raw block-device VLog (--raw-vlogs; Operator rawVLog.enabled / chart storage.rawVlog.enabled) | No filesystem on the value path. Requires root plus SYS_RAWIO, a /dev hostPath mount, and (on GKE COS, per the Operator) a privileged container to pass the cgroup device allowlist. Online backups contain no VLog data in this mode (there is no vlog_active.dat to copy). No raw-vs-XFS latency comparison is published in the main repository, so measure on your hardware before choosing it for latency. |
mkfs.xfs on any device designated for raw VLog use. WAL/segment XFS partitions belong on a separate slice of the disk (xfsPartitionPercent, typically 20% in production) or on the boot disk.
When device paths aren't explicitly configured, the Operator and the Helm chart are documented (in their own repos) to infer them as /dev/nvme0n1 … /dev/nvme(disksPerNode-1)n1. Check that inference against your nodes: on GKE /dev/nvme0n1 can be the boot disk — the server treats any device whose first 64 bytes are zero as blank and writes its superblock there (a GPT disk with an empty boot-code area qualifies), and only refuses devices with other non-VeltrixDB bytes. A wrong path can therefore overwrite another disk's first 4 KB and then its data.
Host kernel settings
The repository ships scripts/sysctl.conf (copy to /etc/sysctl.d/99-veltrixdb.conf and run sysctl --system). These are node-level settings: in Kubernetes they must be applied on the node (node pool config, a privileged init DaemonSet, or the node image), not in the VeltrixDB pod.
| Setting | Value | Why |
|---|---|---|
vm.max_map_count | 262144 | Required at large key counts on cgo builds (the default, including the Docker image). The off-heap native index mmaps each shard array of a page or more: ~3 per shard × 8192 shards, ~25K mappings per engine. With the kernel default of 65530, mmap can fail with ENOMEM and the server panics on insert. New in v1.6 |
fs.file-max | 2097152 | Segment and WAL files per disk plus one descriptor per client connection. |
net.core.rmem_max / wmem_max, net.ipv4.tcp_rmem / tcp_wmem, netdev_max_backlog, tcp_max_syn_backlog | see file | High-concurrency client connections. |
vm.nr_hugepages | 512 | No effect today: nothing in the server enables hugepages (the C++ ART allocator that could use them has no Go call site). Safe to omit. |
RLIMIT_MEMLOCK (ulimit -l, set through limits.conf) only matters if you opt in to the io_uring VLog write bridge (VELTRIXDB_URING_BRIDGE=on|sqpoll; default off). Set the NVMe I/O scheduler to none per device after boot (echo none > /sys/block/nvme0n1/queue/scheduler).
Troubleshooting
| Symptom | Cause / fix |
|---|---|
veltrixdb_vlog_blkdiscard_errors_total > 0 | Raw VLog without CAP_SYS_RAWIO: add SYS_RAWIO to securityContext.capabilities.add. Until then garbage is never trimmed and the device fills. |
Startup fails with raw superblock magic mismatch … refusing to overwrite non-VeltrixDB data | The device holds something else (wrong path, or a filesystem). Fix the --raw-vlogs list; never point it at a formatted device. |
--raw-vlogs (N) must equal --data-dirs (M) | Give exactly one device per data dir, in the same order. |
write /mnt/nvmeN/wal.log: no space left on device | The pod started before the mount-keeper bound real XFS at the PV source path, so kubelet bind-mounted a small tmpfs placeholder instead. Confirm the DaemonSet is Running, then delete the pod so kubelet re-creates the bind mount against the now-formatted disk. |
Pods stuck in Pending | Check that PVs are Available, the provisioner DaemonSet pods are Running, and (on GKE) that the node pool was created with --local-ssd-interface=NVME. |
| Cluster provisions but write throughput is far below benchmark numbers | Verify GKE didn't collapse local SSDs into a single RAID-0 device — this happens automatically if the node pool omits --local-ssd-interface=NVME. |