Admin API Reference
The full HTTP surface exposed on -metrics-addr (default :2112): health probes, Prometheus metrics, and the /admin/* operational endpoints implemented in adminapi/admin.go and cmd/server/admin_cluster.go.
/admin/* is guarded. With no token configured, admin endpoints accept loopback connections only — a non-loopback request gets 403. Starting the server with -admin-token <token> (or the VELTRIX_ADMIN_TOKEN environment variable) requires every /admin/* request, loopback included, to send Authorization: Bearer <token> or X-Admin-Token: <token>; anything else gets 401. The comparison is constant-time. /metrics, /healthz, and /readyz are not guarded, so Prometheus scrapes and Kubernetes probes need no configuration. Several endpoints still mutate live state (/admin/quotas mutates live limits, /admin/migrate mutates on-disk schema), so treat the token as a second layer: on untrusted networks, keep the port behind a network policy / firewall, and preferably mTLS via a sidecar proxy (Envoy or nginx).
Health and metrics endpoints
These are registered directly in cmd/server/main.go, before the storage engine finishes initializing, so liveness checks never time out during a slow WAL replay.
| Method | Path | Purpose |
|---|---|---|
| GET | /healthz | Liveness probe. Always returns 200 ok once the HTTP listener is up — it does not depend on the storage engine being ready. Suitable for a Kubernetes livenessProbe. |
| GET | /readyz | Readiness probe. Returns 503 initializing until the storage engine has finished startup (WAL replay, VLog device open, etc.), then 200 ready. Suitable for a Kubernetes readinessProbe. |
| GET | /metrics | Prometheus exposition endpoint (OpenMetrics-enabled). Backed by veltrixmetrics.NewVeltrixCollector, registered on a dedicated prometheus.Registry. Exposes 60+ metrics covering storage, cache, WAL, VLog GC, replication, and cluster state. |
| GET | /traces | Recent OpenTelemetry-compatible spans from the in-process ring buffer (default retains 2048 spans). Only slow operations (≥50 ms) and errors are guaranteed to appear regardless of the 1% trace sampler. Returned as newline-delimited JSON. |
curl http://localhost:2112/healthz
curl http://localhost:2112/readyz
curl http://localhost:2112/metrics | head -30
curl http://localhost:2112/traces
GET /admin/stats
Returns a full JSON snapshot of engine state: index size, cumulative op counters, cache stats, WAL totals, CDC counters, per-disk VLog stats, and namespace list. This is the endpoint the veltrix status, veltrix compaction, veltrix cache, and veltrix top subcommands poll.
curl http://localhost:2112/admin/stats
Response shape (fields read directly from handleStats in adminapi/admin.go):
| Field | Type | Description |
|---|---|---|
index_keys | int | Live key count in the in-memory index. |
writes_total | uint64 | Cumulative write count. |
reads_total | uint64 | Cumulative read count. |
deletes_total | uint64 | Cumulative delete count. |
atomic_ops_total | uint64 | Cumulative CAS/INCR/DECR/SETNX count. |
audit_dropped_total | uint64 | Audit log entries dropped (e.g., due to a full write queue). |
cache | object | storage.CacheStats — LIRS cache size, capacity, hits, misses, evictions. |
wal_bytes | uint64 | Total bytes written to the WAL across all disks. |
wal_entries | uint64 | Total WAL entry count across all disks. |
cdc_broadcast_total | uint64 | Total CDC events broadcast to subscribers. |
cdc_dropped_total | uint64 | CDC events dropped due to slow consumers. |
cdc_subscribers | int | Current active CDC subscriber count. |
vlogs | array | Per-disk storage.VLogStats — size, live bytes, GC ratio, one entry per disk. |
namespaces | array | storage.NSInfo list — namespace name and key count, one entry per namespace. |
GET /admin/version
Returns the on-disk schema version and whether at-rest encryption is enabled.
curl http://localhost:2112/admin/version
| Field | Type | Description |
|---|---|---|
current_schema_version | int | storage.CurrentSchemaVersion — the schema version this binary writes. |
encryption_enabled | bool | Whether AES-256-GCM at-rest encryption is active on this node. |
POST /admin/checkpoint
Forces an immediate WAL checkpoint across all disks. Synchronous — the HTTP response is delayed until the checkpoint completes.
curl -X POST http://localhost:2112/admin/checkpoint
| Field | Type | Description |
|---|---|---|
status | string | "ok" on success. |
duration_ms | int64 | Wall-clock time the checkpoint took. |
Non-POST requests receive 405 GET or POST is not accepted here — only POST is allowed.
GET / POST /admin/quotas
Reads or mutates per-namespace rate-limit and key-count quotas (token-bucket write throttling plus a hard key-count cap).
GET — list current quotas
curl http://localhost:2112/admin/quotas
Returns a JSON array of storage.QuotaSnapshot objects, one per namespace with a configured limit.
POST — set a namespace quota
Accepts form-encoded parameters (not JSON):
| Form field | Description |
|---|---|
ns | Namespace name to configure. |
writes_per_sec | Sustained write-rate limit (token bucket refill rate). |
burst | Token bucket burst capacity. |
max_keys | Hard cap on key count in the namespace (int64). |
curl -X POST http://localhost:2112/admin/quotas \
-d "ns=tenant_42&writes_per_sec=5000&burst=10000&max_keys=1000000"
Response: {"status": "ok", "ns": "tenant_42"}.
POST /admin/migrate
Runs schema migrations across all keys currently below current_schema_version.
curl -X POST http://localhost:2112/admin/migrate
| Field | Type | Description |
|---|---|---|
migrated | int | Number of records successfully migrated. |
errors | int | Number of records that failed migration. |
target_schema | int | The schema version records were migrated to. |
This is also the endpoint kubectl veltrix migrate calls (see CLI Reference).
GET /admin/cdc
Streams live change-data-capture events as newline-delimited JSON (application/x-ndjson) until the client disconnects, an optional duration elapses, or the subscription is evicted for being a slow consumer.
| Query param | Default | Description |
|---|---|---|
prefix | (all keys) | Only stream events for keys matching this prefix. |
duration_seconds | 0 (unlimited) | Automatically close the stream after N seconds. |
buffer | 1024 | Per-subscriber channel buffer size; slow consumers that fall behind this buffer are evicted. |
curl "http://localhost:2112/admin/cdc?prefix=orders/&duration_seconds=60"
Each streamed line is one JSON-encoded storage.CDCEvent (operation type, key, and related metadata — consumed directly by veltrix cdc-tail and by the repl-ship cross-region shipper).
repl-ship) that is down when an event fires misses it on this stream. As of v1.1.0, the durable /admin/changes catch-up feed closes that gap: repl-ship --checkpoint replays exactly the delta it missed before rejoining the live stream.
GET /admin/changes
v1.1.0 The durable catch-up feed backing cross-region replication (storage.ChangesSince). Where /admin/cdc streams live events, /admin/changes scans the index for entries written at or after a microsecond cursor — so a shipper that persists its "last shipped timestamp" can replay the delta it missed while down, then rejoin the live stream. This is what repl-ship --checkpoint calls on restart.
| Query param | Default | Description |
|---|---|---|
since | 0 | Microsecond timestamp cursor (WriteTimestampUs). The boundary is inclusive (>= since). |
limit | 10000 | Maximum events per page; values ≤ 0 or > 100,000 are reset to 10,000. |
curl "http://localhost:2112/admin/changes?since=1752800000000000&limit=1000"
The response is newline-delimited JSON (application/x-ndjson): one storage.CDCEvent per line (Op "PUT" or "DEL", Key, Value — empty for deletes — and Timestamp in µs), ordered by write timestamp, followed by a single trailer line:
{"Op":"PUT","Key":"orders/1042","Value":"...","Timestamp":1752800000123456}
{"Op":"DEL","Key":"orders/0999","Timestamp":1752800000234567}
{"cursor":1752800000234567,"more":false}
| Trailer field | Type | Description |
|---|---|---|
cursor | int64 (µs) | Resume point for the next page — the timestamp of the last event returned (or the request's since when no events matched). Pass it back as the next request's since. |
more | bool | Whether entries beyond this page matched the scan. |
Semantics worth knowing before building on it:
- The boundary is inclusive (
>= since), and pagination resumes from the last returned event's timestamp — so events sharing the checkpoint's exact microsecond re-ship rather than being lost. Destination applies are last-write-wins Puts/Deletes, so replays are harmless; consumers must be idempotent. - Tombstones are included (
Op:"DEL") — deletes made during shipper downtime reach the remote region too. Expired-TTL entries are skipped (they expire remotely on their own). - The cursor is the single node's clock (
WriteTimestampUs, monotone in practice); the same value is carried asTimestampon the live/admin/cdcpath, so the two feeds share one checkpoint.
POST /admin/backup
Triggers a full or incremental backup from a live engine, writing to a local destination directory on the server's filesystem. Only registered when the server is started with a non-nil BackupEngine (always true for cmd/server).
Request body (JSON):
{
"type": "full",
"dest_dir": "/backup/2026-07-05",
"base_dir": ""
}
| Field | Required | Description |
|---|---|---|
type | No | "full" (default) or "incremental". |
dest_dir | Yes | Local destination directory on the server host. |
base_dir | Only for incremental | Path to the base backup's manifest, used to compute the delta. |
curl -X POST http://localhost:2112/admin/backup \
-H "Content-Type: application/json" \
-d '{"type":"full","dest_dir":"/backup/2026-07-05"}'
Response includes status, type, backup_id, dest_dir, num_disks, duration_ms, and the full manifest object. This is what veltrix backup DEST_DIR calls under the hood.
GET /admin/cluster (alias: /admin/topology)
Returns the live distributed-layer topology: node role, Raft term/leader (in raft mode), replication lag (in replicated mode), partition count, and fencing epoch. Registered separately from the rest of the admin API, in cmd/server/admin_cluster.go, via registerClusterAdmin.
curl http://localhost:2112/admin/cluster
| Field | Type | Description |
|---|---|---|
node_id | string | This node's ID. |
mode | string | standalone, raft, or replicated. |
epoch | uint64 | Partition-map fencing epoch. |
partition_count | uint32 | Number of partitions in the consistent-hash ring. |
consistency | string | Present only in replicated mode — eventual/quorum/strong. |
raft | object | Present only in raft mode: role, term, leader_id, is_leader. |
replication | array | Present only in replicated mode: per-replica node_id, state, last_ack_seq, lag_bytes, lag_ns. |
nodes | array | Every known node: node_id, address, port, state. |
This is the endpoint the bundled cluster-aware client (client.NewClient) calls to bootstrap consistent-hash routing and discover the current Raft leader, and it is what veltrix nodes renders as a table.
GET /admin/ui
Serves the built-in HTML/JS web dashboard (implemented in adminapi/ui.go). Also mounted as the catch-all at /admin/, so any unmatched path under /admin/ falls through to the dashboard.
open http://localhost:2112/admin/ui
Endpoint summary
| Method | Path | Auth | Purpose |
|---|---|---|---|
| GET | /healthz | none | Liveness probe |
| GET | /readyz | none | Readiness probe |
| GET | /metrics | none | Prometheus scrape endpoint |
| GET | /traces | none | Recent slow/error OTel spans |
| GET | /admin/stats | loopback / token | Full engine snapshot |
| GET | /admin/version | loopback / token | Schema version + encryption flag |
| POST | /admin/checkpoint | loopback / token | Force WAL checkpoint |
| GET/POST | /admin/quotas | loopback / token | Read or set per-namespace quotas |
| POST | /admin/migrate | loopback / token | Run schema migrations |
| GET | /admin/cdc | loopback / token | Stream CDC events (NDJSON) |
| GET | /admin/changes | loopback / token | Durable change-feed catch-up (NDJSON + cursor trailer) |
| POST | /admin/backup | loopback / token | Trigger full/incremental backup |
| GET | /admin/cluster, /admin/topology | loopback / token | Cluster topology, Raft/replication state |
| GET | /admin/ui, /admin/* (catch-all) | loopback / token | Web dashboard |