VeltrixDB Docs
v1.16 GitHub FAQ Quickstart

Admin API Reference

The full HTTP surface exposed on -metrics-addr (default :2112): health probes, Prometheus metrics, traces, and the /admin/* operational endpoints implemented in adminapi/admin.go and cmd/server/admin_cluster.go — plus the INFO command on the data port, which is where search-index readiness is reported.

Access control (v1.1.0) /admin/* is guarded (adminapi/guard.go). With no token configured, admin endpoints accept loopback connections only — a non-loopback request gets 403. Starting the server with -admin-token <token> (or the VELTRIX_ADMIN_TOKEN environment variable) requires every /admin/* request, loopback included, to send Authorization: Bearer <token> or X-Admin-Token: <token>; anything else gets 401. The comparison is constant-time. /metrics, /healthz, /readyz and /traces are not guarded, so Prometheus scrapes and Kubernetes probes need no configuration. The veltrix CLI and kubectl-veltrix never send a token: they work against a server without -admin-token from loopback (directly, or through kubectl port-forward), and get 401 from a server that has one. Several endpoints still mutate live state (/admin/quotas mutates live limits, /admin/migrate mutates on-disk schema), so treat the token as a second layer: on untrusted networks, keep the port behind a network policy / firewall, and preferably mTLS via a sidecar proxy (Envoy or nginx).

Health and metrics endpoints

These are registered in cmd/server/main.go. /healthz and /readyz are served from the moment the process starts, so liveness checks never time out during a slow WAL replay; /metrics, /traces and every /admin/* path are added only after the storage engine has opened and return 404 until then.

MethodPathPurpose
GET/healthzLiveness probe. Always returns 200 ok once the HTTP listener is up — it does not depend on the storage engine being ready. Suitable for a Kubernetes livenessProbe.
GET/readyzReadiness probe. Returns 503 initializing until the storage engine has finished startup (WAL replay, VLog device open, etc.), then 200 ready. Returns 503 degraded: disks [...] failed while a disk circuit breaker is tripped (reads keep being served). Does not wait for the search-index rebuild — see INFO. Suitable for a Kubernetes readinessProbe.
GET/metricsPrometheus exposition endpoint (OpenMetrics-enabled). Backed by veltrixmetrics.NewVeltrixCollector, registered on a dedicated prometheus.Registry. Exposes about 80 metric families covering storage, cache, WAL, VLog GC, replication, and cluster state.
GET/tracesRecent OpenTelemetry-compatible spans from the in-process ring buffer (default retains 2048 spans). Only slow operations (≥50 ms) and errors are guaranteed to appear regardless of the 1% trace sampler. Returned as newline-delimited JSON. Query params: limit=N (at most N spans) and error=true (only spans with an error status).
curl http://localhost:2112/healthz
curl http://localhost:2112/readyz
curl http://localhost:2112/metrics | head -30
curl "http://localhost:2112/traces?limit=50&error=true"

GET /admin/stats

Returns a full JSON snapshot of engine state: index size, cumulative op counters, cache stats, WAL totals, CDC counters, per-disk VLog stats, and namespace list. This is the endpoint the veltrix status, compaction, cache, wal, replication, cdc and top subcommands poll (since v1.14, replication reads /admin/cluster instead). Methods other than GET get 405.

curl http://localhost:2112/admin/stats

Response shape (fields read directly from handleStats in adminapi/admin.go):

FieldTypeDescription
index_keysintLive key count in the in-memory index.
writes_totaluint64Cumulative write count.
reads_totaluint64Cumulative read count.
deletes_totaluint64Cumulative delete count.
atomic_ops_totaluint64Cumulative CAS/INCR/DECR/SETNX count.
audit_dropped_totaluint64Audit log entries dropped (e.g., due to a full write queue).
cacheobjectstorage.CacheStats — LIRS cache size, capacity, hits, misses, evictions.
wal_bytesuint64Total bytes written to the WAL across all disks.
wal_entriesuint64Total WAL entry count across all disks.
cdc_broadcast_totaluint64Total CDC events broadcast to subscribers.
cdc_dropped_totaluint64CDC events dropped due to slow consumers.
cdc_subscribersintCurrent active CDC subscriber count.
vlogsarrayPer-disk storage.VLogStats — size, live bytes, GC ratio, one entry per disk.
namespacesarraystorage.NSInfo list — namespace name and key count, one entry per namespace.

GET /admin/version

Returns the on-disk schema version and whether at-rest encryption is enabled.

curl http://localhost:2112/admin/version
FieldTypeDescription
current_schema_versionintstorage.CurrentSchemaVersion — the schema version this binary writes.
encryption_enabledboolWhether AES-256-GCM at-rest encryption is active on this node.

POST /admin/checkpoint

Forces an immediate WAL checkpoint across all disks. Synchronous — the HTTP response is delayed until the checkpoint completes.

curl -X POST http://localhost:2112/admin/checkpoint
FieldTypeDescription
statusstring"ok" on success.
duration_msint64Wall-clock time the checkpoint took.

Any method other than POST gets 405 POST only; a failed checkpoint returns 500 with the error text.

v1.16 and earlier: writes after a checkpoint are not crash-safe On every release v1.0–v1.16 this endpoint (and POST /admin/backup) leaves the WAL flusher writing to the unlinked old wal.log: on standalone and replicated nodes every later write is lost on a crash until the node’s next clean restart. Restart the node cleanly after calling it. Fixed on main after v1.16 — ships in the next release the checkpoint swaps the flusher onto the new file and is safe under writes. See Backup & Restore.

GET / POST /admin/quotas

Reads or mutates per-namespace rate-limit and key-count quotas (token-bucket write throttling plus a hard key-count cap).

GET — list current quotas

curl http://localhost:2112/admin/quotas

Returns a JSON array of storage.QuotaSnapshot objects, one per namespace with a configured limit. The struct has no JSON tags, so the fields are capitalised: Namespace, WritesPerSec, BurstWrites, MaxKeys, KeyCount, TokensLeft. Methods other than GET/POST get 405 GET or POST.

POST — set a namespace quota

Accepts form-encoded parameters (not JSON):

Form fieldDescription
nsNamespace name to configure.
writes_per_secSustained write-rate limit (token bucket refill rate).
burstToken bucket burst capacity.
max_keysHard cap on key count in the namespace (int64).
curl -X POST http://localhost:2112/admin/quotas \
  -d "ns=tenant_42&writes_per_sec=5000&burst=10000&max_keys=1000000"

Response: {"status": "ok", "ns": "tenant_42"}. Omitted or non-numeric fields are set to 0; the request always replaces the namespace's whole limit.

POST /admin/migrate

Runs schema migrations across all keys currently below current_schema_version.

curl -X POST http://localhost:2112/admin/migrate
FieldTypeDescription
migratedintNumber of records successfully migrated.
errorsintNumber of records that failed migration.
target_schemaintThe schema version records were migrated to.

This is also the endpoint kubectl veltrix migrate calls (see CLI Reference).

GET /admin/cdc

Streams live change-data-capture events as newline-delimited JSON (application/x-ndjson) until the client disconnects, an optional duration elapses, or the subscription is evicted for being a slow consumer.

Query paramDefaultDescription
prefix(all keys)Only stream events for keys matching this prefix.
duration_seconds0 (unlimited)Automatically close the stream after N seconds.
buffer1024Per-subscriber channel buffer size; slow consumers that fall behind this buffer are evicted.
curl "http://localhost:2112/admin/cdc?prefix=orders/&duration_seconds=60"

Each streamed line is one JSON-encoded storage.CDCEvent: Op, Key, Value (base64, as Go encodes []byte; null for deletes) and Timestamp (µs). It is consumed by the repl-ship cross-region shipper; veltrix cdc-tail reads it too; in v1.12 its formatter looks for lowercase field names, so its op/key columns print — (Fixed in v1.14 it decodes these fields; see CLI Reference).

The live CDC stream is in-process only Events are held in an in-memory channel per subscriber, so a consumer (e.g. repl-ship) that is down when an event fires misses it on this stream. As of v1.1.0, the durable /admin/changes catch-up feed closes that gap: repl-ship --checkpoint replays exactly the delta it missed before rejoining the live stream.

GET /admin/changes

v1.1.0 The durable catch-up feed backing cross-region replication (storage.ChangesSince). Where /admin/cdc streams live events, /admin/changes scans the index for entries written at or after a microsecond cursor — so a shipper that persists its "last shipped timestamp" can replay the delta it missed while down, then rejoin the live stream. This is what repl-ship --checkpoint calls on restart.

Query paramDefaultDescription
since0Microsecond timestamp cursor (WriteTimestampUs). The boundary is inclusive (>= since).
limit10000Maximum events per page; values ≤ 0 or > 100,000 are reset to 10,000.
curl "http://localhost:2112/admin/changes?since=1752800000000000&limit=1000"

The response is newline-delimited JSON (application/x-ndjson): one storage.CDCEvent per line (Op "PUT" or "DEL", Key, Value — base64, null for deletes — and Timestamp in µs), ordered by write timestamp, followed by a single trailer line. Methods other than GET get 405.

{"Op":"PUT","Key":"orders/1042","Value":"eyJ0b3RhbCI6NDJ9","Timestamp":1752800000123456}
{"Op":"DEL","Key":"orders/0999","Value":null,"Timestamp":1752800000234567}
{"cursor":1752800000234567,"more":false}
Trailer fieldTypeDescription
cursorint64 (µs)Resume point for the next page — the timestamp of the last event returned (or the request's since when no events matched). Pass it back as the next request's since.
moreboolWhether entries beyond this page matched the scan.

Semantics worth knowing before building on it:

POST /admin/backup

Triggers a full or incremental backup from the running engine, written by the server process to a local destination directory on the server's filesystem. This (or veltrix backup DEST_DIR, which calls it) is the way to back up a running server: the offline veltrixdb-backup tool opens its own engine on the data dirs and must only be pointed at a stopped server's directories — against a live server it truncated the live wal.log and vlog_active.dat in testing. The endpoint is always registered by cmd/server. On v1.16 and earlier, restart the node cleanly after an online backup — writes made after it are lost on a crash until then (see above).

Request body (JSON):

{
  "type": "full",
  "dest_dir": "/backup/2026-07-05",
  "base_dir": ""
}
FieldRequiredDescription
typeNo"full" (default) or "incremental".
dest_dirYesLocal destination directory on the server host.
base_dirOnly for incrementalDirectory of the base backup (it must contain that backup's manifest.json), used to compute the delta.
curl -X POST http://localhost:2112/admin/backup \
  -H "Content-Type: application/json" \
  -d '{"type":"full","dest_dir":"/backup/2026-07-05"}'
curl -X POST http://localhost:2112/admin/backup \
  -d '{"type":"incremental","dest_dir":"/backup/2026-07-05-1500","base_dir":"/backup/2026-07-05"}'

Response includes status, type, backup_id, dest_dir (and base_dir for incremental), num_disks, duration_ms, and the full manifest object. Errors: 400 for a bad body, a missing dest_dir, a missing or unreadable base_dir, or an unknown type; 500 if the backup itself fails; 405 for non-POST. veltrix backup DEST_DIR sends {"type":"full","dest_dir":DEST_DIR}.

GET /admin/cluster (alias: /admin/topology)

Returns the live distributed-layer topology: node role, Raft term/leader (in raft mode), replication lag (in replicated mode), partition count, and fencing epoch. Registered separately from the rest of the admin API, in cmd/server/admin_cluster.go, via registerClusterAdmin.

curl http://localhost:2112/admin/cluster
FieldTypeDescription
node_idstringThis node's ID.
modestringstandalone, raft, or replicated.
epochuint64Partition-map fencing epoch.
partition_countuint32Number of partitions in the consistent-hash ring.
consistencystringPresent only in replicated mode — eventual/quorum/strong.
raftobjectPresent only in raft mode: role, term, leader_id, is_leader.
replicationarrayPresent only in replicated mode: per-replica node_id, state, last_ack_seq, lag_bytes, lag_ns. Fixed in v1.14 lag_ns is the age of the oldest write the replica has not acked (0 when caught up), and two fields are added: last_ack_unix_nano and, while the replica is FAILED, next_probe_unix_nano. New on main after v1.16 — ships in the next release resync_pending (a change-feed catch-up is pending or running) and delete_gap (the replica was behind by more than the tombstone retention; deletes may be missing on it).
nodesarrayEvery known node: node_id, address, port, rack (omitted when unset; see -rack-id), state.

veltrix nodes renders this endpoint as a table. The same JSON, on one line, is returned by the TOPOLOGY text command on the data port — that is what the bundled cluster-aware client (client.NewClient) uses to build its consistent-hash ring and find the Raft leader, so it needs no admin access. Non-GET methods get 405.

GET / POST /admin/cluster/members, DELETE /admin/cluster/members/{id} New on main after v1.16 — ships in the next release

Online membership changes (cmd/server/membership.go), with the same auth as every other /admin/* endpoint. Needs --mode=raft or --mode=replicated (otherwise 400). Any node accepts a change and forwards it to the raft leader or the replicated-mode coordinator over the transfer listener, which requires --cluster-secret-file or --cluster-mtls. Procedures: membership runbook.

curl http://localhost:2112/admin/cluster/members
curl -X POST http://localhost:2112/admin/cluster/members \
  -d '{"id":"n4","addr":"10.0.0.14:9000","wait_ms":60000}'
curl -X DELETE 'http://localhost:2112/admin/cluster/members/n2?wait_ms=120000'
curl -X DELETE 'http://localhost:2112/admin/cluster/members/r2?force=true'   # replicated: dead node, no drain
GET fieldDescription
mode, node_idMode and this node's ID.
version, committedRaft: index of the configuration entry (0 = the bootstrap --peers) and whether it is committed. Replicated: the membership version.
leader / coordinatorRaft leader, or the replicated-mode coordinator (lowest member ID).
self_statusvoter, learner, member, joining, draining or removed.
membersEach member's id, client addr and role (voter / learner in raft, member in replicated).

POST body: id, addr (the node's client --addr) and optional wait_ms. DELETE query: force=true (replicated: remove without draining — writes the dead node acknowledged but never replicated are lost) and wait_ms. Answers: 200 done; 202 raft learner added, promotion still pending; 400 bad request or last voter; 404 unknown id; 409 duplicate id, a change already in progress, a concurrent replicated change, or a failed drain; 503 no reachable quorum, no leader, or coordinator unreachable. CLI: veltrix cluster members | add-node ID@HOST:PORT | remove-node ID [--force] [--wait-ms N].

INFO (data port, not HTTP)

INFO is a text-protocol command on -addr (default :9000), not an HTTP endpoint, but it is the one place a node reports whether its search indexes are ready. With -auth-config it needs the admin permission. The reply is one line of key=value pairs:

$ printf 'INFO\nQUIT\n' | nc 127.0.0.1 9000
keys=1203 writes=5120 reads=880 deletes=4 cache_hits=812 cache_misses=68 hit_rate=0.92 wal_flushes=311 compaction_runs=0 version=5128 search_ready=0 search_loaded=4100/9000 disk[0]=./veltrixdb-data:12.4MB
BYE
FieldDescription
keysLive key count in the index.
writes, reads, deletesCumulative operation counters.
cache_hits, cache_misses, hit_rateLIRS cache counters and hit rate (2 decimals).
wal_flushes, compaction_runsWAL fdatasync flushes and compaction runs.
versionThe engine's current write-version counter.
search_readyNew in v1.11 1 once the startup rebuild of the vector / text indexes from the persisted @vec/ and @txt/ keys has finished. While it is 0, vector / text / hybrid searches are refused with search indexes are still rebuilding after restart (loaded x of y; …) unless the server runs with -search-allow-partial; KV traffic, QUERY and IDXQUERY are not affected. /readyz does not wait for this.
search_loadedNew in v1.11 Present only while search_ready=0: loaded/total keys rebuilt so far.
disk[N]Per disk: path:segmentMB.
FAILED_DISKSPresent only when a disk has failed: the failed disk indexes.

The binary INFO frame (0x05) returns a shorter string — keys, writes, reads, cache_hit_rate, wal_flushes — without the search fields, so use the text command to poll readiness. veltrix-admin stats prints the text reply as a table.

GET /admin/ui

Serves the built-in HTML/JS web dashboard (implemented in adminapi/ui.go). Also mounted as the catch-all at /admin/, so any unmatched path under /admin/ falls through to the dashboard.

open http://localhost:2112/admin/ui

Endpoint summary

MethodPathAuthPurpose
GET/healthznoneLiveness probe
GET/readyznoneReadiness probe
GET/metricsnonePrometheus scrape endpoint
GET/tracesnoneRecent slow/error OTel spans
text cmdINFO on -addrRBAC admin (with -auth-config)Counters, search_ready / search_loaded, disks
GET/admin/statsloopback / tokenFull engine snapshot
GET/admin/versionloopback / tokenSchema version + encryption flag
POST/admin/checkpointloopback / tokenForce WAL checkpoint
GET/POST/admin/quotasloopback / tokenRead or set per-namespace quotas
POST/admin/migrateloopback / tokenRun schema migrations
GET/admin/cdcloopback / tokenStream CDC events (NDJSON)
GET/admin/changesloopback / tokenDurable change-feed catch-up (NDJSON + cursor trailer)
POST/admin/backuploopback / tokenTrigger full/incremental backup
GET/admin/cluster, /admin/topologyloopback / tokenCluster topology, Raft/replication state
GET/POST/DELETE/admin/cluster/members, /admin/cluster/members/{id}loopback / tokenNew on main after v1.16 — ships in the next release List, add and remove cluster members
GET/admin/ui, /admin/* (catch-all)loopback / tokenWeb dashboard
Profiling is not on this port New in v1.7 /debug/pprof/ (profile?seconds=N, trace?seconds=N, heap, goroutine, …) is served only when the server is started with -pprof-addr, on its own listener and mux — never on -metrics-addr. It has no auth; bind it to loopback. New in v1.16 /debug/pprof/mutex and /debug/pprof/block are empty unless -pprof-mutex-fraction / -pprof-block-rate are set. See Server Configuration.