Monitoring & Observability
VeltrixDB exposes a Prometheus scrape endpoint, liveness/readiness probes, and a web dashboard, all served from the same admin HTTP port alongside 60+ Prometheus metrics.
Endpoints
curl http://localhost:2112/metrics # Prometheus scrape endpoint
curl http://localhost:2112/healthz # liveness probe
curl http://localhost:2112/readyz # readiness probe
open http://localhost:2112/admin/ui # web dashboardAll of these are served on the admin/metrics port (2112 by default), separate from the client data port (9000 by default). /healthz and /readyz are suitable as Kubernetes liveness and readiness probes respectively; the bundled Helm chart wires them in for you.
Key metrics to watch
VeltrixDB exposes 60+ Prometheus metrics under the veltrixdb_ prefix. The core set to alert on:
| Metric | Healthy value |
|---|---|
veltrixdb_writes_total / veltrixdb_reads_total | Growing under load |
veltrixdb_cache_hits_total / veltrixdb_cache_misses_total | Hit rate > 90% |
veltrixdb_storage_write_admission_throttles_total | 0 |
veltrixdb_vlog_garbage_ratio | < 0.30 |
veltrixdb_vlog_gc_emergency_runs_total | 0 |
The Grafana dashboards shipped with the project (see below) additionally chart a wider set of series observed in the metrics registry and dashboard queries, including:
| Metric | What it shows |
|---|---|
veltrixdb_storage_writes_total / veltrixdb_storage_reads_total / veltrixdb_storage_deletes_total | Per-node operation counters (as scraped with the node label) |
veltrixdb_storage_write_latency_seconds / veltrixdb_storage_read_latency_seconds / veltrixdb_storage_delete_latency_seconds | Latency histograms — source for P50/P95/P99 via histogram_quantile |
veltrixdb_storage_wal_flushes_total | WAL group-commit fdatasync rate — compare against writes/s for amortization ratio |
veltrixdb_vlog_writes_total | VLog append rate |
veltrixdb_cache_hit_rate / veltrixdb_cache_size_bytes / veltrixdb_cache_max_size_bytes | LIRS cache hit rate and RAM used vs. capacity |
veltrixdb_cache_evictions_total (labeled type=lirs|ttl|defrag) | Eviction pressure by cause |
veltrixdb_vlog_file_bytes | VLog size per NVMe disk — also the numerator for capacity planning against PVC size |
veltrixdb_vlog_gc_runs_total / veltrixdb_vlog_gc_bytes_total | Background GC activity |
veltrixdb_storage_index_size_keys | In-memory index size (keys held in DRAM) |
veltrixdb_cluster_nodes_total | Cluster topology size, for multi-node deployments |
Useful PromQL for day-to-day operation:
rate(veltrixdb_writes_total[1m]) # write throughput
rate(veltrixdb_reads_total[1m]) # read throughput
rate(veltrixdb_cache_hits_total[5m]) /
(rate(veltrixdb_cache_hits_total[5m]) + rate(veltrixdb_cache_misses_total[5m])) # hit ratio
rate(veltrixdb_writes_total[1m]) / rate(veltrixdb_wal_flushes_total[1m]) # WAL amortization
histogram_quantile(0.99, rate(veltrixdb_storage_read_latency_seconds_bucket[5m])) # P99 readDisk-failure auto-degrade
VeltrixDB detects failing disks automatically with a per-disk consecutive-error breaker: 5 consecutive I/O errors on a disk's WAL/VLog trip the breaker and mark the disk FAILED. Any successful I/O before the fifth error resets the streak, so transient hiccups don't trip it. Once tripped:
- Writes routed to that disk fail fast with
ErrDiskFailed("disk marked FAILED — node is degraded, retry on another replica") instead of hitting the dead device's timeouts. Clients should treat it as a retriable node-degraded condition and retry against another replica. - Reads still try the device — data may be partially readable — and are served from cache when possible.
- GC and compaction skip the failed disk.
/readyzreturns503with a body likedegraded: disks [2] failed, so orchestrators (Kubernetes readiness probes) stop routing new traffic to the node while reads keep being served.INFOreportsFAILED_DISKS=[2], and the engine'sDiskFailuresmetric counts breaker trips.
Admin API
The same admin port also exposes a small JSON control-plane API, documented in full in the Admin API reference:
curl http://localhost:2112/admin/stats # engine snapshot (JSON)
curl http://localhost:2112/admin/version # schema version
curl -X POST http://localhost:2112/admin/checkpoint # force WAL checkpoint
curl http://localhost:2112/admin/cdc?prefix= # live CDC events (NDJSON)
curl http://localhost:2112/admin/cluster # topology: role, raft term/leader, peers, epoch, replica lagGrafana dashboards
The repository ships provisioned Grafana dashboards under grafana/ — a Prometheus datasource definition, a dashboard provider config, and dashboard JSON including a full-stack operational dashboard (veltrixdb_dashboard.json) covering write/read throughput, write/read/delete latency percentiles, LIRS cache hit rate and evictions, per-disk VLog size and garbage ratio, and GC run rate. A separate CI-focused dashboard (ci_dashboard.json) is also included. These are ready to mount into a Grafana provisioning directory alongside the Prometheus scrape config.
The Kubernetes deployment additionally installs a ServiceMonitor for automatic Prometheus scraping and a set of 22 PrometheusRule alerts covering slow disk, GC emergency, corruption detection, and replication lag conditions out of the box.