Limitations & Roadmap
These are real gaps in VeltrixDB today. We'd rather you know them upfront than discover them in production.
Honest limitations
Current as of v1.16 (2026-10-08, the latest release), checked against the server source. Gaps that v1.14 (2026-10-05) and v1.15 fixed are listed under Fixed in v1.14 and v1.15 — they still apply to an older install until you upgrade. Gaps already fixed on the main branch but not yet released are listed under Fixed on main after v1.16; on v1.16 and earlier they still apply.
POST /admin/backup, veltrix backup) or POST /admin/checkpoint leaves the WAL flusher writing to an unlinked copy of wal.log. On standalone and replicated nodes, every write acknowledged afterwards is lost on a crash, power loss or SIGKILL until the node’s next clean restart (SIGTERM), which rewrites wal.log from the in-memory index; raft mode recovers those writes from the raft log. Workaround: after each online backup or checkpoint, restart the node cleanly as soon as you can. Fixed on main after v1.16 — ships in the next release see Backup & Restore.Product and ecosystem
nc-compatible) or one of the native SDKs; the command surface (PUT/GET/DEL, atomic ops) is close to Redis's. Roadmap: yes; nothing in the code implements it yet.bench/compare holds a go-ycsb runner with Aerospike and ScyllaDB drivers and a vector benchmark, but the Aerospike / ScyllaDB paths are compile-checked only and nothing has been run side by side. The October 2026 numbers are VeltrixDB-only, mostly on one macOS laptop. Treat any cross-database claim as unmeasured.Consistency and clustering
QUERY and IDXQUERY do fan out.) Since v1.15 a running server never moves keys between nodes — raft applies the whole log on every node, and replicated mode sends every write to every peer — so a scan covers the whole keyspace, but only as fresh as that node: a raft follower can lag the leader, and a replicated node may not have applied a write accepted elsewhere yet. On v1.14 and earlier a node failure with the default --auto-rebalance=true moved keys off a node, after which its scans reflect only the keys it still holds (see Fixed in v1.14 and v1.15). Workaround: in raft mode, --linearizable-reads (followers redirect, and every read passes the ReadIndex fence); on v1.14 and earlier, --auto-rebalance=false, or fan out and merge client-side.replicated mode has no single-writer ordering; replicas apply ops in arrival order (version vectors are reserved but unused, no conflict resolution runs). Anti-entropy re-sends un-acked ops only; it never compares keyspaces. --consistency defaults to eventual. Raft and replicated mode are mutually exclusive. On v1.13 and earlier a FAILED replica also stays failed until the primary restarts — fixed in v1.14; the fix keeps its catch-up state in memory, so a replica that was down when its primary restarted still misses the writes made before that restart. See Fixed in v1.14 and v1.15. Two clients writing one key through different nodes at the same time can leave the nodes holding different values for it permanently — still open. On v1.16 and earlier a replica that was down while its primary restarted never gets the writes and deletes made before that restart (Fixed on main after v1.16 — ships in the next release the acknowledgement watermark is persisted; a replica away longer than the tombstone retention is fully resynced and flagged FULL RESYNC, and a delete may survive on it). Workaround: raft mode for linearizable writes, or write each key through one node; on v1.13 and earlier, restart the primary after a replica outage and verify the replica.INCRs doubled), and SETIF version tokens differ between nodes. Fixed on main after v1.16 — ships in the next release see Fixed on main after v1.16. During a rolling upgrade from v1.16, a v1.16 follower can only receive a single-shot snapshot (≤ 256 MB, TTLs lost, deletes not applied) and SETIF / guarded TXN stay nondeterministic until every node is upgraded.--peers at startup; there is no runtime join/drain/remove command, and adding a node means restarting every node with the new --peers list. New on main after v1.16 — ships in the next release nodes are added and removed online (veltrix cluster add-node / remove-node, /admin/cluster/members; raft adds a learner and promotes it, replicated mode drains a node before removing it — see the runbook). What remains: one change at a time; node-to-node membership traffic, --join and every replicated-mode change need --cluster-secret-file or --cluster-mtls; replicated mode has no consensus, so a member that misses a change pulls it later (a window of disagreement during a partition); adding a replicated node copies every key from every member; a dead replicated node can only be force-removed, losing writes it acknowledged but never replicated; every node must run the new build before the first change; the Kubernetes operator still scales by restarting pods with a new --peers, which stops working once a change was made online. A node failure is a state change, not a membership change, so since v1.15 it never migrates keys. (On v1.13 and earlier tombstone reaping is also not coordinated with replica watermarks — fixed in v1.14.)QUERY and IDXQUERY runs on all non-failed nodes, so cost grows with the cluster. A peer that does not answer within --search-timeout-ms fails the request (naming it) until the failure detector marks it failed, unless --search-allow-partial.Vector and text search
search_ready=0 in INFO; /readyz does not wait). Budget restart time for large namespaces. Graph nodes, ids and upper layers stay on the Go heap even with GRAPH disk (~280–400 B per vector).Security and data path
AUTH or TTL expiry), never fills the actor field, drops records when its channel is full, and is not hash-chained. On v1.13 and earlier it also covers only single-key PUT/DEL (not batches, transactions or atomic ops) — fixed in v1.14. See Fixed in v1.14 and v1.15. See also Security./admin/cdc never blocks a writer: a subscriber whose buffer is full loses events and is disconnected (on v1.13 and earlier after 3 drops in total; since v1.14, 3 consecutive drops). repl-ship is last-write-wins with no back-pressure to the source. On v1.13 and earlier the stream also carries only single-key PUT/DEL — MPUT, coalesced pipelined PUTs, transactions and atomic ops reach a shipper only through the index-backed /admin/changes catch-up (last state only); fixed in v1.14. See Fixed in v1.14 and v1.15.veltrixdb-backup must never run against a live server's data dirs (it can truncate them) — use POST /admin/backup or veltrix backup; v1.14 and later refuse instead (data-dir lock, see Fixed in v1.14 and v1.15). On v1.16 and earlier writes made after an online backup are not crash-safe until a clean restart (see the warning above). With --raw-vlogs backups contain no VLog data, and backup does not pause GC. See Backup & Restore.go_* or process_* metrics (so the shipped restart alert cannot fire) and no search or raft metrics (New on main after v1.16 — ships in the next release served, with 24 alerts checked against what a raft and a replicated server serve; still missing: process_* other than process_start_time_seconds is Linux only, search metrics count requests on the coordinating node only, CDC and audit drops are in /admin/stats only). On v1.13 and earlier replication_lag_nanoseconds / cluster_partition_migrations_total are also always 0 — fixed in v1.14. See Fixed in v1.14 and v1.15. See Monitoring.Engine and front-end
--net=cpp|uring|poll serves only the binary protocol's PUT GET DEL PING MPUT MGET, only in standalone mode, and refuses --auth-config; the text protocol, AUTH, TLS and every other command stay on the default --net=go. In CI measurements it lowered read P50 but had worse read P99 and ~12% lower batch-write throughput than the Go front-end, so it is opt-in and not faster overall. Workaround: keep --net=go unless you have measured a benefit on your hardware with scripts/net-bench.sh.mlockall and the eBPF GC throttle exist under cpp/ but have no Go call site, so they do not run. The VLog read path is Go pread. The io_uring VLog write bridge is opt-in (VELTRIXDB_URING_BRIDGE, default off).Fixed in v1.14 and v1.15 v1.14–v1.15
These were gaps in v1.13 and earlier (fixed in v1.14) or v1.14 and earlier (fixed in v1.15). Each item says which release fixed it: v1.14 (2026-10-05) or v1.15 (2026-10-07). v1.16 (2026-10-08) is the latest release. Until you upgrade, the older behaviour described in each item still applies.
- Data-dir lock. v1.13 and earlier: never point the offline
veltrixdb-backup(orveltrix-repair) at a running server's data dirs — it opens a second engine on the live files, and in testing that produced an empty backup and truncated the livewal.logandvlog_active.dat. Fixed in v1.14: every engine holds an exclusive lock on<data-dir>/LOCK, so a second engine —veltrixdb-backup,veltrix-repairor a second server — is refused withErrDataDirLockedand the live files are not touched;veltrixdb-backuptells you to stop the server or usePOST /admin/backup/veltrix backup.restore/restore-pitrtake the same lock. - Atomic ops and the value transform. v1.13 and earlier:
CAS/INCR/DECR/SETNXstore the raw value (plaintext even under--encrypt-at-rest) and misread a compressed or encrypted value on a cache miss — do not mix them with encryption at rest on the same keys. Fixed in v1.14: they compress and encrypt likePUT. Values written by older builds stay plaintext until their next update. - TTLs survive restarts; expired keys are not served from the cache. v1.13 and earlier: the WAL carries no TTL, so after any restart a TTL'd key comes back without an expiry (and one that had expired comes back), and a cached key could be served after it expired. Fixed in v1.14: a TTL'd key's WAL record carries its absolute expiry (binary record version 2, or an 11th field in the text format); replay restores it and drops keys that already expired; checkpoints, backups and PITR archives carry it; a cache hit past the deadline is not served. Rolling back to v1.13 or older: run once with
--wal-format=text-legacyand stop cleanly (TTL'd keys become immortal on the older build); v1.14 and later read the records directly, no step. - MGET returns empty values. v1.13 and earlier: an empty value written by
MPUTreads as missing inMGET. Fixed in v1.14:MGETagrees withGET. - CDC and audit cover every write path. v1.13 and earlier: both see only single-key
PUT/DEL, and a CDC subscriber is closed after 3 drops in total. Fixed in v1.14:MPUT, pipelined PUTs,TXN, Raft batch apply and the atomic ops all emit one CDC event and one audit record per key (atomic ops are audited asCAS/INCR/DECR/SETNX, including outcomes that wrote nothing); a subscriber is evicted only after 3 consecutive drops; the audit log is drained and fsynced on shutdown. DELremoves a record's vectors and text documents. v1.13 and earlier: deleting a KV record leaves@vec/<ns>/<id>and@txt/<ns>/<id>in place, so searches can still return the id — issueVDEL/TDELas well. Fixed in v1.14:DELof a record also deletes its vectors and documents in every namespace (crash-safe: the startup rebuild removes any left behind). Vectors and documents that never had a record are untouched; TTL expiry does not cascade.- The Raft read barrier covers every read. v1.13 and earlier: only GET and MGET wait for a new leader's no-op (and use the ReadIndex fence with
--linearizable-reads); namespace, hash,RANGE/SCANCURandGETVERreads use local state. Fixed in v1.14: every read on both protocols — namespace, hash, list, set,RANGE/SCANCUR,VER/GETVER,IDXQUERY,QUERYand all searches — passes the barrier. - Replication recovers on its own. v1.13 and earlier: a
FAILEDreplica stays failed until the primary restarts, a node markedFAILEDnever gets back fromRECOVERINGtoACTIVE, a full replication write queue drops the op, and tombstones are reaped after the 24 h grace period regardless of replicas. Fixed in v1.14:FAILEDreplicas are re-probed with backoff (1 s doubling to 30 s) and caught up throughSYNC_PENDINGbefore returning toSYNC; aRECOVERINGnode returns toACTIVEafter 2 good pings; a full queue keeps the op for anti-entropy; tombstone reaping waits for replica watermarks (capped at 2× the grace period). - Auto-rebalance no longer deletes keys on a node failure. v1.14 and earlier: with the default
--auto-rebalance=true, a node markedFAILEDmakes every node send each key to its single ring owner and delete it locally, outside the Raft log and the replication stream and ignoring the replication factor. Killing the leader of a loaded 3-node raft cluster (20,000 keys) left 5,435 and 7,875 keys on the survivors. Run clusters with--auto-rebalance=falseuntil you upgrade. Fixed in v1.15: migration never runs in--mode=raft(the flag is a logged no-op,/transfer/keysanswers HTTP 403), never on a node state change, and in replicated mode a key is deleted from a node only after every current replica acknowledged it. See Cluster & Consensus. - Raft persistence is incremental. v1.14 and earlier: every Raft flush rewrites and fsyncs the whole retained log into
raft_state.gob, and a follower fsyncs eachAppendEntriesunder the node lock, so heartbeats and votes wait behind it. Fixed in v1.15: an append-onlyraft_log.datplusraft_meta.dat(term and vote), one write and one fsync of the new records per flush, outside the node lock; followers acknowledge only durable entries, and the leader counts itself toward a commit only once its own fsync covers the entry; a follower's commit index is capped at the last entry the request verified; a deposed leader clears its leader ID (no moreMOVEDloop after an election). On one laptop (3 local nodes, not a cluster result) raft load went from 70 to 1,046 inserts/s.raft_state.gobis migrated on first start; a rollback to v1.14 or earlier needs the node's data wiped and re-synced (details). New flag--raft-pipeline(envVELTRIX_RAFT_PIPELINE, default off): up to 8 AppendEntries in flight per follower, sent before the leader's own fsync. Durability is the same either way; on 3 local nodes sharing one SSD it held load to ~600 inserts/s against ~1,050 with it off, so it stays off until measured on separate hosts. - Replication metrics. v1.13 and earlier:
veltrixdb_replication_lag_nanosecondsandveltrixdb_cluster_partition_migrations_totalare always 0. Fixed in v1.14: lag is the age of the oldest write a replica has not acked, and migrations count acknowledged key batches;veltrixdb_replication_conflict_resolutions_totalandveltrixdb_replication_vector_clock_updates_total(always 0) are removed. - CLI. v1.13 and earlier:
veltrix statusalways prints NOT READY,cdc-tailprints blank fields, flags after the command are ignored, andveltrixandkubectl-veltrixsend no admin token, so they work only against servers started without--admin-token. Fixed in v1.14: readiness comes from the/readyzstatus code,cdc-taildecodes the events,--admin-token/VELTRIX_ADMIN_TOKENonveltrix,kubectl-veltrixandrepl-ship(where it aliases--src-token), flags work after the command,kubectl veltrix quota-settakes--burst, andcache/status/compaction/top/quotas/replication/tracesread the right fields. - Go client. v1.13 and earlier:
BinaryConn.Put/PipelineandClient.Put(WithTTL)drop the TTL, andGetreturnsnil, nilfor any server error. Fixed in v1.14: the TTL is sent,Getsurfaces server errors andClient.GetfollowsMOVED;WithConsistencyis deprecated (the protocol has no consistency field). - Startup banner no longer claims an ART index or io_uring scheduler (neither runs).
Fixed on main after v1.16 Fixed on main after v1.16 — ships in the next release
These are fixed or added on the main branch and ship in the next release. On v1.16 and earlier the older behaviour described in each item still applies.
- Writes after an online backup or checkpoint survive a crash. v1.0–v1.16:
POST /admin/backup,veltrix backupandPOST /admin/checkpointleave the WAL flusher on the unlinked oldwal.log, so on standalone and replicated nodes every later write is lost on a crash until the next clean restart. Fixed: the checkpoint swaps the flusher onto the new file between two group-commit batches (details). - Raft snapshots stream and have no size cap. v1.16 and earlier: snapshots over 256 MB are skipped (the log grows without bound), capture stalls Apply for the whole scan, TTLs are dropped, keys deleted while a follower lagged survive its restore, and secondary-index definitions are not carried. Fixed:
raft_snapshot.dat(migrated fromraft_snapshot.gob), chunked InstallSnapshot (--raft-snapshot-chunk-kb),--raft-snapshot-threshold, restore equals the snapshot. Laptop, 1,000,000 keys: slowest write during a capture 2.76 s → 28 ms; follower catch-up 42 s → 2.6 s (details). - Raft applies each entry once across restarts. v1.16 and earlier: every restart re-applies the retained log —
INCR/DECRdoubled, list pushes repeated, guardedTXNs fail. Fixed with a durable applied index; restarts with a snapshot also skip the O(keys) restore (details). - Raft Apply is deterministic. v1.16 and earlier:
SETIF/ guardedTXNversion tokens are each node’s own timestamp, so replicas diverge. Fixed: versions come from the leader’s clock. Do not rely onSETIFuntil every node is upgraded (details). - Replicated-mode catch-up survives a primary restart (
replication_state.json); a replica away longer than the tombstone retention is flaggedFULL RESYNC(details). Concurrent writers on different nodes can still diverge. - A raft follower answers a rejected batch write with
MOVED. v1.16 and earlier: anMPUTsent to a follower — including every TTL’dPUT, which the SDKs send as a one-entryMPUT— failed with per-entry error bytes and no leader address. Fixed: the whole batch is answered with theMOVED <addr> <leaderId>error frame, which every SDK already raises as its MOVED error. - Online membership (new):
/admin/cluster/members,veltrix cluster,--join,--advertise-addr,--force-new-cluster, raft learners and pre-vote, the replicated drain (runbook). - Monitoring (new):
go_*/process_*, search, raft and snapshot, and replication-recovery metrics; 24 alert rules, two of which could not fire on v1.16 (Monitoring). - Testing (new): a Jepsen-style consistency harness and rolling-upgrade / downgrade tests against v1.16, both nightly (details).
Shipped in v1.11–v1.12 New
v1.13 (2026-10-01) was documentation and test fixes over v1.12; v1.16 (2026-10-08) is the latest release, and what v1.14 and v1.15 fixed is listed under Fixed in v1.14 and v1.15. These used to be roadmap items or gaps:
- Vector quantization —
VCREATE … QUANT int8(~⅓ of float32 RAM, re-ranked against the stored float32 vectors) andQUANT pqproduct quantization (v1.12), plusGRAPH diskto move layer-0 graph edges into a memory-mapped file (v1.12). Measured at 768-dim: 3.1× (int8) to 8.9× (PQ + disk graph) less heap per vector. - BM25 full-text search (
TSET/TDEL/TSEARCH) and hybrid search (HSEARCH, weighted reciprocal rank fusion) with record filters and secondary-index-backed=filters. - Cluster search — vector / text / hybrid search,
QUERYandIDXQUERYfan out to every node and merge (BM25 with global corpus statistics), failing closed unless--search-allow-partial; derived keys route with their record so filters evaluate locally. - HMAC cluster secret (
--cluster-secret-file/VELTRIXDB_CLUSTER_SECRET, ≥ 16 bytes) signing all transfer-listener traffic (v1.11). - Adaptive group commit (
--group-commit=adaptive, default, v1.12): the flush window is an upper bound, so a lone writer is synced at once (emulated 300 µs sync: P50 16 ms → 0.45 ms) and 64 concurrent writers went 4.0K → 37.5K writes/s. - Raft read barrier for GET / MGET after leader failover (v1.12) — extended to every other read path in v1.14 (see above).
- Soak and chaos tests (v1.12) — nightly 20-minute search soak against an oracle, 10 SIGKILL/restart cycles checking every acknowledged write, and GloVe-100 recall gates.
- Searches are refused during the startup index rebuild instead of silently missing vectors (
search_readyinINFO).
Shipped in v1.6–v1.8
- Off-heap native index on every cgo build (including the Docker image);
VELTRIXDB_INDEX=mapopts out. Needsvm.max_map_count ≥ 262144at large key counts. - Binary WAL by default, with
--wal-format=textfor rollback; keys containing|or a newline no longer truncate replay. - Ordered-index overwrite skip (server batch writes 1.45M → 3.54M keys/s on macOS).
- io_uring VLog bridge made opt-in (
VELTRIXDB_URING_BRIDGE, default off). - Experimental C++ network front-end (
--net,--net-threads) with deferred disk reads and a shutdown stage-latency report. --pprof-addr,loadtest --proto,scripts/net-bench.sh, andveltrix-repair.
Shipped in v1.1.0
The following were listed as limitations before the v1.1.0 "distributed-completeness" release and are now implemented — they no longer belong on the list above:
- Rack/zone-aware replica placement — start each node with
--rack-id <zone>and partition placement never puts two copies of a partition in the same failure domain (unless the replication factor exceeds the number of distinct racks, in which case placement still spans every rack available). Rack labels propagate via gossip, and rackless clusters keep the exact previous placement. New in v1.1.0, so less battle-tested than the rest of the placement path. See Cluster & Consensus. - Admin API access control —
/admin/*now defaults to loopback-only, with an optional bearer token (--admin-token) for remote access. See Security. - HNSW vector index — replaced the brute-force scan (reworked in 2026-09; current recall/latency figures are on Vector & Hybrid Search, e.g. GloVe-100 recall@10 0.842 at ef 64 and 0.953 at ef 256).
- SDK atomics everywhere —
CAS/INCR/DECR/SETNXadded to the Go, Python, Node.js, and Java clients (previously Rust/C++ only). See the SDK docs. - Auto-rebalance — node join/leave/failure triggers ring rebalance and physical key migration; opt out with
--auto-rebalance=false. Fixed in v1.15 failures no longer trigger it and raft mode never migrates (see above); with static membership a running server no longer migrates at all (New on main after v1.16 — ships in the next release replicated mode migrates again on an online add or remove, RF-aware and put-if-absent). - Linearizable reads in raft mode — with
--linearizable-reads, GET runs the Raft ReadIndex fence: quorum-confirmed, never stale (every read since v1.14). - List and set data types —
LPUSH/RPUSH/LPOP/RPOP/LLEN/LRANGEandSADD/SREM/SISMEMBER/SMEMBERS/SCARD. - Distributed routing for all data types — namespace, hash-field, vector, secondary-index, and list/set ops go through the coordinator in raft and replicated modes. Secondary-index definitions now also replicate in replicated mode (as
@idxdef/keys); the local-read gaps above are what's left. - Durable cross-DC catch-up — the
/admin/changesfeed plusrepl-ship --checkpoint: a restarted shipper replays exactly the delta it missed before rejoining the live CDC stream. See Replication & CDC. - Ordered range scans —
RANGEandSCANCURrun over an ordered skiplist of every live key (~90 B/key); point-lookup-only deployments can turn it off with--disable-ordered-index. - Raft log snapshots — once the retained log reaches 8192 entries the node writes
raft_snapshot.goband compacts the log; lagging followers catch up viaInstallSnapshot. (Fixed on main after v1.16 — ships in the next release streamed toraft_snapshot.dat, no size cap, chunked transfer.) - PBKDF2 password hashing — PBKDF2-HMAC-SHA256 (210,000 iterations, per-user random salt) with legacy-hash fallback and a load-time deprecation warning. See Security.
What's coming
From the project changelog's "What's coming" section:
- RESP protocol (Redis compatibility) — point Redis clients at VeltrixDB without code changes.
- Managed cloud offering.
- Multi-region replication with automatic failover —
repl-shipships changes across regions today; failover is manual.