Admin API Reference
The full HTTP surface exposed on -metrics-addr (default :2112): health probes, Prometheus metrics, traces, and the /admin/* operational endpoints implemented in adminapi/admin.go and cmd/server/admin_cluster.go — plus the INFO command on the data port, which is where search-index readiness is reported.
/admin/* is guarded (adminapi/guard.go). With no token configured, admin endpoints accept loopback connections only — a non-loopback request gets 403. Starting the server with -admin-token <token> (or the VELTRIX_ADMIN_TOKEN environment variable) requires every /admin/* request, loopback included, to send Authorization: Bearer <token> or X-Admin-Token: <token>; anything else gets 401. The comparison is constant-time. /metrics, /healthz, /readyz and /traces are not guarded, so Prometheus scrapes and Kubernetes probes need no configuration. The veltrix CLI and kubectl-veltrix never send a token: they work against a server without -admin-token from loopback (directly, or through kubectl port-forward), and get 401 from a server that has one. Several endpoints still mutate live state (/admin/quotas mutates live limits, /admin/migrate mutates on-disk schema), so treat the token as a second layer: on untrusted networks, keep the port behind a network policy / firewall, and preferably mTLS via a sidecar proxy (Envoy or nginx).
Health and metrics endpoints
These are registered in cmd/server/main.go. /healthz and /readyz are served from the moment the process starts, so liveness checks never time out during a slow WAL replay; /metrics, /traces and every /admin/* path are added only after the storage engine has opened and return 404 until then.
| Method | Path | Purpose |
|---|---|---|
| GET | /healthz | Liveness probe. Always returns 200 ok once the HTTP listener is up — it does not depend on the storage engine being ready. Suitable for a Kubernetes livenessProbe. |
| GET | /readyz | Readiness probe. Returns 503 initializing until the storage engine has finished startup (WAL replay, VLog device open, etc.), then 200 ready. Returns 503 degraded: disks [...] failed while a disk circuit breaker is tripped (reads keep being served). Does not wait for the search-index rebuild — see INFO. Suitable for a Kubernetes readinessProbe. |
| GET | /metrics | Prometheus exposition endpoint (OpenMetrics-enabled). Backed by veltrixmetrics.NewVeltrixCollector, registered on a dedicated prometheus.Registry. Exposes about 80 metric families covering storage, cache, WAL, VLog GC, replication, and cluster state. |
| GET | /traces | Recent OpenTelemetry-compatible spans from the in-process ring buffer (default retains 2048 spans). Only slow operations (≥50 ms) and errors are guaranteed to appear regardless of the 1% trace sampler. Returned as newline-delimited JSON. Query params: limit=N (at most N spans) and error=true (only spans with an error status). |
curl http://localhost:2112/healthz
curl http://localhost:2112/readyz
curl http://localhost:2112/metrics | head -30
curl "http://localhost:2112/traces?limit=50&error=true"
GET /admin/stats
Returns a full JSON snapshot of engine state: index size, cumulative op counters, cache stats, WAL totals, CDC counters, per-disk VLog stats, and namespace list. This is the endpoint the veltrix status, compaction, cache, wal, replication, cdc and top subcommands poll (since v1.14, replication reads /admin/cluster instead). Methods other than GET get 405.
curl http://localhost:2112/admin/stats
Response shape (fields read directly from handleStats in adminapi/admin.go):
| Field | Type | Description |
|---|---|---|
index_keys | int | Live key count in the in-memory index. |
writes_total | uint64 | Cumulative write count. |
reads_total | uint64 | Cumulative read count. |
deletes_total | uint64 | Cumulative delete count. |
atomic_ops_total | uint64 | Cumulative CAS/INCR/DECR/SETNX count. |
audit_dropped_total | uint64 | Audit log entries dropped (e.g., due to a full write queue). |
cache | object | storage.CacheStats — LIRS cache size, capacity, hits, misses, evictions. |
wal_bytes | uint64 | Total bytes written to the WAL across all disks. |
wal_entries | uint64 | Total WAL entry count across all disks. |
cdc_broadcast_total | uint64 | Total CDC events broadcast to subscribers. |
cdc_dropped_total | uint64 | CDC events dropped due to slow consumers. |
cdc_subscribers | int | Current active CDC subscriber count. |
vlogs | array | Per-disk storage.VLogStats — size, live bytes, GC ratio, one entry per disk. |
namespaces | array | storage.NSInfo list — namespace name and key count, one entry per namespace. |
GET /admin/version
Returns the on-disk schema version and whether at-rest encryption is enabled.
curl http://localhost:2112/admin/version
| Field | Type | Description |
|---|---|---|
current_schema_version | int | storage.CurrentSchemaVersion — the schema version this binary writes. |
encryption_enabled | bool | Whether AES-256-GCM at-rest encryption is active on this node. |
POST /admin/checkpoint
Forces an immediate WAL checkpoint across all disks. Synchronous — the HTTP response is delayed until the checkpoint completes.
curl -X POST http://localhost:2112/admin/checkpoint
| Field | Type | Description |
|---|---|---|
status | string | "ok" on success. |
duration_ms | int64 | Wall-clock time the checkpoint took. |
Any method other than POST gets 405 POST only; a failed checkpoint returns 500 with the error text.
POST /admin/backup) leaves the WAL flusher writing to the unlinked old wal.log: on standalone and replicated nodes every later write is lost on a crash until the node’s next clean restart. Restart the node cleanly after calling it. Fixed on main after v1.16 — ships in the next release the checkpoint swaps the flusher onto the new file and is safe under writes. See Backup & Restore.GET / POST /admin/quotas
Reads or mutates per-namespace rate-limit and key-count quotas (token-bucket write throttling plus a hard key-count cap).
GET — list current quotas
curl http://localhost:2112/admin/quotas
Returns a JSON array of storage.QuotaSnapshot objects, one per namespace with a configured limit. The struct has no JSON tags, so the fields are capitalised: Namespace, WritesPerSec, BurstWrites, MaxKeys, KeyCount, TokensLeft. Methods other than GET/POST get 405 GET or POST.
POST — set a namespace quota
Accepts form-encoded parameters (not JSON):
| Form field | Description |
|---|---|
ns | Namespace name to configure. |
writes_per_sec | Sustained write-rate limit (token bucket refill rate). |
burst | Token bucket burst capacity. |
max_keys | Hard cap on key count in the namespace (int64). |
curl -X POST http://localhost:2112/admin/quotas \
-d "ns=tenant_42&writes_per_sec=5000&burst=10000&max_keys=1000000"
Response: {"status": "ok", "ns": "tenant_42"}. Omitted or non-numeric fields are set to 0; the request always replaces the namespace's whole limit.
POST /admin/migrate
Runs schema migrations across all keys currently below current_schema_version.
curl -X POST http://localhost:2112/admin/migrate
| Field | Type | Description |
|---|---|---|
migrated | int | Number of records successfully migrated. |
errors | int | Number of records that failed migration. |
target_schema | int | The schema version records were migrated to. |
This is also the endpoint kubectl veltrix migrate calls (see CLI Reference).
GET /admin/cdc
Streams live change-data-capture events as newline-delimited JSON (application/x-ndjson) until the client disconnects, an optional duration elapses, or the subscription is evicted for being a slow consumer.
| Query param | Default | Description |
|---|---|---|
prefix | (all keys) | Only stream events for keys matching this prefix. |
duration_seconds | 0 (unlimited) | Automatically close the stream after N seconds. |
buffer | 1024 | Per-subscriber channel buffer size; slow consumers that fall behind this buffer are evicted. |
curl "http://localhost:2112/admin/cdc?prefix=orders/&duration_seconds=60"
Each streamed line is one JSON-encoded storage.CDCEvent: Op, Key, Value (base64, as Go encodes []byte; null for deletes) and Timestamp (µs). It is consumed by the repl-ship cross-region shipper; veltrix cdc-tail reads it too; in v1.12 its formatter looks for lowercase field names, so its op/key columns print — (Fixed in v1.14 it decodes these fields; see CLI Reference).
repl-ship) that is down when an event fires misses it on this stream. As of v1.1.0, the durable /admin/changes catch-up feed closes that gap: repl-ship --checkpoint replays exactly the delta it missed before rejoining the live stream.
GET /admin/changes
v1.1.0 The durable catch-up feed backing cross-region replication (storage.ChangesSince). Where /admin/cdc streams live events, /admin/changes scans the index for entries written at or after a microsecond cursor — so a shipper that persists its "last shipped timestamp" can replay the delta it missed while down, then rejoin the live stream. This is what repl-ship --checkpoint calls on restart.
| Query param | Default | Description |
|---|---|---|
since | 0 | Microsecond timestamp cursor (WriteTimestampUs). The boundary is inclusive (>= since). |
limit | 10000 | Maximum events per page; values ≤ 0 or > 100,000 are reset to 10,000. |
curl "http://localhost:2112/admin/changes?since=1752800000000000&limit=1000"
The response is newline-delimited JSON (application/x-ndjson): one storage.CDCEvent per line (Op "PUT" or "DEL", Key, Value — base64, null for deletes — and Timestamp in µs), ordered by write timestamp, followed by a single trailer line. Methods other than GET get 405.
{"Op":"PUT","Key":"orders/1042","Value":"eyJ0b3RhbCI6NDJ9","Timestamp":1752800000123456}
{"Op":"DEL","Key":"orders/0999","Value":null,"Timestamp":1752800000234567}
{"cursor":1752800000234567,"more":false}
| Trailer field | Type | Description |
|---|---|---|
cursor | int64 (µs) | Resume point for the next page — the timestamp of the last event returned (or the request's since when no events matched). Pass it back as the next request's since. |
more | bool | Whether entries beyond this page matched the scan. |
Semantics worth knowing before building on it:
- The boundary is inclusive (
>= since), and pagination resumes from the last returned event's timestamp — so events sharing the checkpoint's exact microsecond re-ship rather than being lost. Destination applies are last-write-wins Puts/Deletes, so replays are harmless; consumers must be idempotent. - Tombstones are included (
Op:"DEL") — deletes made during shipper downtime reach the remote region too. Expired-TTL entries are skipped (they expire remotely on their own). - The cursor is the single node's clock (
WriteTimestampUs, monotone in practice); the same value is carried asTimestampon the live/admin/cdcpath, so the two feeds share one checkpoint.
POST /admin/backup
Triggers a full or incremental backup from the running engine, written by the server process to a local destination directory on the server's filesystem. This (or veltrix backup DEST_DIR, which calls it) is the way to back up a running server: the offline veltrixdb-backup tool opens its own engine on the data dirs and must only be pointed at a stopped server's directories — against a live server it truncated the live wal.log and vlog_active.dat in testing. The endpoint is always registered by cmd/server. On v1.16 and earlier, restart the node cleanly after an online backup — writes made after it are lost on a crash until then (see above).
Request body (JSON):
{
"type": "full",
"dest_dir": "/backup/2026-07-05",
"base_dir": ""
}
| Field | Required | Description |
|---|---|---|
type | No | "full" (default) or "incremental". |
dest_dir | Yes | Local destination directory on the server host. |
base_dir | Only for incremental | Directory of the base backup (it must contain that backup's manifest.json), used to compute the delta. |
curl -X POST http://localhost:2112/admin/backup \
-H "Content-Type: application/json" \
-d '{"type":"full","dest_dir":"/backup/2026-07-05"}'
curl -X POST http://localhost:2112/admin/backup \
-d '{"type":"incremental","dest_dir":"/backup/2026-07-05-1500","base_dir":"/backup/2026-07-05"}'
Response includes status, type, backup_id, dest_dir (and base_dir for incremental), num_disks, duration_ms, and the full manifest object. Errors: 400 for a bad body, a missing dest_dir, a missing or unreadable base_dir, or an unknown type; 500 if the backup itself fails; 405 for non-POST. veltrix backup DEST_DIR sends {"type":"full","dest_dir":DEST_DIR}.
GET /admin/cluster (alias: /admin/topology)
Returns the live distributed-layer topology: node role, Raft term/leader (in raft mode), replication lag (in replicated mode), partition count, and fencing epoch. Registered separately from the rest of the admin API, in cmd/server/admin_cluster.go, via registerClusterAdmin.
curl http://localhost:2112/admin/cluster
| Field | Type | Description |
|---|---|---|
node_id | string | This node's ID. |
mode | string | standalone, raft, or replicated. |
epoch | uint64 | Partition-map fencing epoch. |
partition_count | uint32 | Number of partitions in the consistent-hash ring. |
consistency | string | Present only in replicated mode — eventual/quorum/strong. |
raft | object | Present only in raft mode: role, term, leader_id, is_leader. |
replication | array | Present only in replicated mode: per-replica node_id, state, last_ack_seq, lag_bytes, lag_ns. Fixed in v1.14 lag_ns is the age of the oldest write the replica has not acked (0 when caught up), and two fields are added: last_ack_unix_nano and, while the replica is FAILED, next_probe_unix_nano. New on main after v1.16 — ships in the next release resync_pending (a change-feed catch-up is pending or running) and delete_gap (the replica was behind by more than the tombstone retention; deletes may be missing on it). |
nodes | array | Every known node: node_id, address, port, rack (omitted when unset; see -rack-id), state. |
veltrix nodes renders this endpoint as a table. The same JSON, on one line, is returned by the TOPOLOGY text command on the data port — that is what the bundled cluster-aware client (client.NewClient) uses to build its consistent-hash ring and find the Raft leader, so it needs no admin access. Non-GET methods get 405.
GET / POST /admin/cluster/members, DELETE /admin/cluster/members/{id} New on main after v1.16 — ships in the next release
Online membership changes (cmd/server/membership.go), with the same auth as every other /admin/* endpoint. Needs --mode=raft or --mode=replicated (otherwise 400). Any node accepts a change and forwards it to the raft leader or the replicated-mode coordinator over the transfer listener, which requires --cluster-secret-file or --cluster-mtls. Procedures: membership runbook.
curl http://localhost:2112/admin/cluster/members
curl -X POST http://localhost:2112/admin/cluster/members \
-d '{"id":"n4","addr":"10.0.0.14:9000","wait_ms":60000}'
curl -X DELETE 'http://localhost:2112/admin/cluster/members/n2?wait_ms=120000'
curl -X DELETE 'http://localhost:2112/admin/cluster/members/r2?force=true' # replicated: dead node, no drain
| GET field | Description |
|---|---|
mode, node_id | Mode and this node's ID. |
version, committed | Raft: index of the configuration entry (0 = the bootstrap --peers) and whether it is committed. Replicated: the membership version. |
leader / coordinator | Raft leader, or the replicated-mode coordinator (lowest member ID). |
self_status | voter, learner, member, joining, draining or removed. |
members | Each member's id, client addr and role (voter / learner in raft, member in replicated). |
POST body: id, addr (the node's client --addr) and optional wait_ms. DELETE query: force=true (replicated: remove without draining — writes the dead node acknowledged but never replicated are lost) and wait_ms. Answers: 200 done; 202 raft learner added, promotion still pending; 400 bad request or last voter; 404 unknown id; 409 duplicate id, a change already in progress, a concurrent replicated change, or a failed drain; 503 no reachable quorum, no leader, or coordinator unreachable. CLI: veltrix cluster members | add-node ID@HOST:PORT | remove-node ID [--force] [--wait-ms N].
INFO (data port, not HTTP)
INFO is a text-protocol command on -addr (default :9000), not an HTTP endpoint, but it is the one place a node reports whether its search indexes are ready. With -auth-config it needs the admin permission. The reply is one line of key=value pairs:
$ printf 'INFO\nQUIT\n' | nc 127.0.0.1 9000
keys=1203 writes=5120 reads=880 deletes=4 cache_hits=812 cache_misses=68 hit_rate=0.92 wal_flushes=311 compaction_runs=0 version=5128 search_ready=0 search_loaded=4100/9000 disk[0]=./veltrixdb-data:12.4MB
BYE
| Field | Description |
|---|---|
keys | Live key count in the index. |
writes, reads, deletes | Cumulative operation counters. |
cache_hits, cache_misses, hit_rate | LIRS cache counters and hit rate (2 decimals). |
wal_flushes, compaction_runs | WAL fdatasync flushes and compaction runs. |
version | The engine's current write-version counter. |
search_ready | New in v1.11 1 once the startup rebuild of the vector / text indexes from the persisted @vec/ and @txt/ keys has finished. While it is 0, vector / text / hybrid searches are refused with search indexes are still rebuilding after restart (loaded x of y; …) unless the server runs with -search-allow-partial; KV traffic, QUERY and IDXQUERY are not affected. /readyz does not wait for this. |
search_loaded | New in v1.11 Present only while search_ready=0: loaded/total keys rebuilt so far. |
disk[N] | Per disk: path:segmentMB. |
FAILED_DISKS | Present only when a disk has failed: the failed disk indexes. |
The binary INFO frame (0x05) returns a shorter string — keys, writes, reads, cache_hit_rate, wal_flushes — without the search fields, so use the text command to poll readiness. veltrix-admin stats prints the text reply as a table.
GET /admin/ui
Serves the built-in HTML/JS web dashboard (implemented in adminapi/ui.go). Also mounted as the catch-all at /admin/, so any unmatched path under /admin/ falls through to the dashboard.
open http://localhost:2112/admin/ui
Endpoint summary
| Method | Path | Auth | Purpose |
|---|---|---|---|
| GET | /healthz | none | Liveness probe |
| GET | /readyz | none | Readiness probe |
| GET | /metrics | none | Prometheus scrape endpoint |
| GET | /traces | none | Recent slow/error OTel spans |
| text cmd | INFO on -addr | RBAC admin (with -auth-config) | Counters, search_ready / search_loaded, disks |
| GET | /admin/stats | loopback / token | Full engine snapshot |
| GET | /admin/version | loopback / token | Schema version + encryption flag |
| POST | /admin/checkpoint | loopback / token | Force WAL checkpoint |
| GET/POST | /admin/quotas | loopback / token | Read or set per-namespace quotas |
| POST | /admin/migrate | loopback / token | Run schema migrations |
| GET | /admin/cdc | loopback / token | Stream CDC events (NDJSON) |
| GET | /admin/changes | loopback / token | Durable change-feed catch-up (NDJSON + cursor trailer) |
| POST | /admin/backup | loopback / token | Trigger full/incremental backup |
| GET | /admin/cluster, /admin/topology | loopback / token | Cluster topology, Raft/replication state |
| GET/POST/DELETE | /admin/cluster/members, /admin/cluster/members/{id} | loopback / token | New on main after v1.16 — ships in the next release List, add and remove cluster members |
| GET | /admin/ui, /admin/* (catch-all) | loopback / token | Web dashboard |
/debug/pprof/ (profile?seconds=N, trace?seconds=N, heap, goroutine, …) is served only when the server is started with -pprof-addr, on its own listener and mux — never on -metrics-addr. It has no auth; bind it to loopback. New in v1.16 /debug/pprof/mutex and /debug/pprof/block are empty unless -pprof-mutex-fraction / -pprof-block-rate are set. See Server Configuration.