Monitoring & Observability

VeltrixDB exposes a Prometheus scrape endpoint (about 85 veltrixdb_* metric families), liveness/readiness probes, a slow-trace endpoint and a web dashboard, all on the same admin HTTP port.

Endpoints

curl http://localhost:2112/metrics   # Prometheus scrape endpoint
curl http://localhost:2112/healthz   # liveness probe ("ok")
curl http://localhost:2112/readyz    # readiness probe ("ready" / 503)
curl http://localhost:2112/traces    # recent sampled + slow (≥ 50 ms) spans, NDJSON
open http://localhost:2112/admin/ui  # web dashboard (loopback-only unless -admin-token)

All of these are served on the admin/metrics port (-metrics-addr, :2112 by default), separate from the client data port (9000 by default). /metrics, /healthz and /readyz are never authenticated. /readyz answers 503 initializing until the engine is open and 503 degraded: disks [N] failed when a disk breaker has tripped; it does not wait for WAL replay or the search-index rebuild — check INFO for search_ready=1 before sending searches to a restarted node.

Key metrics to watch

All metric families carry a node label (per-disk ones add disk). The core set to alert on:

Other useful series (every name below is defined in metrics/prometheus.go):

Useful PromQL for day-to-day operation:

rate(veltrixdb_storage_writes_total[1m])                            # write throughput
rate(veltrixdb_storage_reads_total[1m])                             # read throughput

rate(veltrixdb_cache_hits_total[5m]) /
  (rate(veltrixdb_cache_hits_total[5m]) + rate(veltrixdb_cache_misses_total[5m])) # hit ratio

rate(veltrixdb_storage_writes_total[1m]) / rate(veltrixdb_storage_wal_flushes_total[1m])  # WAL amortization

histogram_quantile(0.99, rate(veltrixdb_storage_read_latency_seconds_bucket[5m])) # P99 read

Search metrics New on main after v1.16 — ships in the next release

Every mode. The coordinating node counts each client vector (VSEARCH, VSEARCHX), text (TSEARCH) and hybrid (HSEARCH) request once, fan-out included; peer-to-peer /internal/search calls are not counted. Recording a request allocates nothing.

Raft and snapshot metrics New on main after v1.16 — ships in the next release

--mode=raft only, read from the raft node at scrape time. On v1.16 and earlier read the term and leader from GET /admin/cluster or veltrix nodes.

Replication recovery metrics New on main after v1.16 — ships in the next release

--mode=replicated only. Per replica, the same state is in GET /admin/cluster as replication[].resync_pending and replication[].delete_gap. See Replication & CDC.

Disk-failure auto-degrade

VeltrixDB detects failing disks automatically with a per-disk consecutive-error breaker: 5 consecutive I/O errors on a disk's WAL/VLog trip the breaker and mark the disk FAILED. Any successful I/O before the fifth error resets the streak, so transient hiccups don't trip it. Once tripped:

• Writes routed to that disk fail fast with ErrDiskFailed ("disk marked FAILED — node is degraded, retry on another replica") instead of hitting the dead device's timeouts. Clients should treat it as a retriable node-degraded condition and retry against another replica.
• Reads still try the device — data may be partially readable — and are served from cache when possible.
• GC and compaction skip the failed disk.
• /readyz returns 503 with a body like degraded: disks [2] failed, so orchestrators (Kubernetes readiness probes) stop routing new traffic to the node while reads keep being served.
• INFO reports FAILED_DISKS=[2], and the engine's internal DiskFailures counter records breaker trips (it is not exported as a Prometheus series; alert on /readyz or INFO).

Admin API

The same admin port also exposes a small JSON control-plane API, documented in full in the Admin API reference:

curl    http://localhost:2112/admin/stats       # engine snapshot (JSON)
curl    http://localhost:2112/admin/version     # schema version
curl -X POST http://localhost:2112/admin/checkpoint  # force WAL checkpoint
curl    http://localhost:2112/admin/cdc?prefix= # live CDC events (NDJSON)
curl    http://localhost:2112/admin/cluster     # topology: role, raft term/leader, peers, epoch, replica lag
curl    http://localhost:2112/admin/cluster/members  # membership (new on main after v1.16)

Grafana dashboards

The repository ships provisioned Grafana dashboards under grafana/ — a Prometheus datasource and dashboard provider under grafana/provisioning/, an operational dashboard (veltrixdb_dashboard.json: throughput, latency percentiles, cache hit rate and evictions, per-disk VLog size and garbage ratio, GC), a larger veltrixdb_complete_dashboard.json, and a CI dashboard (ci_dashboard.json). The runtime, process, system and disk panels query veltrixdb_runtime_* / veltrixdb_process_* / veltrixdb_system_* / veltrixdb_disk_* series from the hardware collector, which v1.16 and earlier do not register, so they stay empty against those servers (New on main after v1.16 — ships in the next release registered).

The repository also ships monitoring/servicemonitor.yaml (scrapes port metrics every 15 s on pods labelled app.kubernetes.io/name=veltrixdb in namespace veltrixdb) and monitoring/prometheus-rules.yaml, a PrometheusRule with 10 alerts in v1.16:

New on main after v1.16 — ships in the next release the rules file has 24 rules (the Helm chart's prometheusRule.extendedAlerts groups match), and a test starts a raft and a replicated server and fails if a rule names a metric neither serves:

• storage: the eight above. VeltrixDBVLogGCStalled now compares the target's worst disk (on v1.16 it joined a per-node counter with a per-disk ratio and could not fire).
• process: disk over 85% full (node-exporter); VeltrixDBProcessRestarted fires while process_start_time_seconds is less than 5 minutes old (also on a first start).
• search: rebuild stuck (not ready, no keys loaded for 30 minutes), partial results, rebuilding / incomplete errors for 10 minutes, P99 over 1 s.
• raft: snapshot captures failing, snapshot transfers failing (leader), snapshot install or restore failing (follower), log not compacted (retained log over 4× the threshold for 15 minutes), apply lag over 10,000 entries, term churn, no leader.
• replication: full resync / delete gap (critical: deletes may be missing on a replica), replication_state.json write errors, a replica catch-up pending for 30 minutes.

Profiling New in v1.7

Start the server with -pprof-addr=127.0.0.1:6060 to serve CPU, heap, goroutine and execution-trace profiles on a separate listener (never the metrics port):

go tool pprof http://127.0.0.1:6060/debug/pprof/profile?seconds=30   # CPU
go tool pprof http://127.0.0.1:6060/debug/pprof/heap                 # heap
curl -o trace.out 'http://127.0.0.1:6060/debug/pprof/trace?seconds=5' # execution trace

New in v1.16 /debug/pprof/mutex and /debug/pprof/block stay empty unless you also pass -pprof-mutex-fraction N (sample 1 in N mutex contention events; 5–100 is typical) and -pprof-block-rate NS (record blocking events of at least NS nanoseconds, e.g. 10000). Both default to 0 (off) and are ignored without -pprof-addr.

When the experimental C++ network front-end (-net=cpp|uring|poll) is in use, the server also logs a per-stage latency report at shutdown (stages iter, cb_enter, exec, put_sched, put_exec, wake, defer_sched, defer_exec; log2 µs buckets).