Data Model

VeltrixDB is a byte-oriented key-value store: keys and values are opaque byte strings, and every higher-level feature — TTLs, hash fields, namespaces, atomic counters, transactions — is layered on top of that single primitive.

Keys and values

A key is an arbitrary byte string and a value is an arbitrary byte string. VeltrixDB does not interpret or type-check value contents at the storage layer — encoding (JSON, protobuf, msgpack, raw binary) is entirely up to the application. Internally, every key maps to exactly one 64-byte IndexEntry in the in-memory index (the "Index Vault"), which stores the disk offset, segment ID, value size, write timestamp, TTL deadline, and a version counter. The value itself lives in the per-disk append-only Value Log (VLog); the index never holds value bytes, which is what keeps random-access lookups to a single cache-line fetch plus, at worst, one NVMe read.

Per-key TTL

Every PUT accepts an optional TTL in seconds. A key written with a TTL carries an absolute expiry timestamp (TTLExpiryUs) in its index entry; a TTL of 0 or a negative value means the key is immortal (no expiry). Expiry is checked lazily on read (a GET against an expired key returns not-found) and is also swept by a background TTL scanner so expired entries don't linger indefinitely and get reclaimed by VLog garbage collection.

Hash fields with per-field TTL

A hash key groups multiple named fields under one logical key, and — unlike a single flat value — each field carries its own independent TTL. VeltrixDB implements this without a separate on-disk structure: a hash field is stored as a normal index entry whose internal key is key + "\x01" + field (a reserved separator byte distinct from the namespace separator). Because each field is just another entry in the same sharded index, per-field expiry, per-field overwrite, and per-field storage cost all fall out of the existing PUT/GET/DELETE machinery — no new WAL format or index type was needed.

The operations are:

Namespaces

A namespace isolates a slice of the keyspace under a prefix. Internally, a namespaced key is stored as namespace + "\x00" + key — the null byte is a safe separator because it never appears in normal UTF-8 keys — so namespaces reuse the same PUT/GET/DELETE paths and on-disk formats as plain keys; there is no separate namespace table. This makes namespaces a convenient way to give each tenant (or each logical dataset) its own key-count quota and write-rate limit without any additional storage engine.

Namespace operations:

Namespace quotas are enforced by a QuotaManager: a token-bucket rate limit covers PUT/DEL/CAS/INCR/SETNX (reads are never rate-limited), and a separate key-count cap rejects new-key writes once a namespace is full — overwrites of existing keys and deletes still succeed over quota, so a tenant can always delete its way back under the limit. Quota state is in-memory; key counts are rebuilt from the index on WAL replay after a restart.

Lists & deques

A list (added in v1.1.0) is a double-ended queue with Redis-style semantics. Like hash fields and namespaces, lists introduce no new on-disk format: each element is a regular KV entry whose internal key is key + "\x02" + seq, where seq is an 8-byte big-endian sequence number (big-endian so lexicographic scans return list order), and a meta entry at key + "\x02!" stores the list's [8B head][8B tail] bounds. A push allocates the next sequence outward from the appropriate end; a pop consumes inward. Multi-key push/pop operations (element + meta) serialize on a striped per-list mutex, and the element is written before the meta commits the new bounds — a crash in between leaves an orphan element outside the committed range, which is invisible to reads and overwritten by the next push at that sequence.

$ nc localhost 9000
RPUSH jobs render-frame-1
1
RPUSH jobs render-frame-2
2
LPUSH jobs urgent-fix
3
LRANGE jobs 0 -1
urgent-fix
render-frame-1
render-frame-2
END
LPOP jobs
urgent-fix
LLEN jobs
2

Sets

A set (added in v1.1.0) stores unique string members under one key. Each member is a regular KV entry whose internal key is key + "\x03" + member with an empty value — membership is simply the existence of the entry, so SISMEMBER is a single index lookup and SMEMBERS/SCARD are prefix scans. As with lists, there is no new WAL format or index type.

$ nc localhost 9000
SADD tags urgent
1
SADD tags billing
1
SADD tags urgent
0
SISMEMBER tags urgent
1
SCARD tags
2
SREM tags billing
1
SMEMBERS tags
urgent
END

Vector search

VeltrixDB stores float32 embedding vectors and answers approximate nearest-neighbour queries by cosine similarity. As of v1.1.0, queries run on an in-memory HNSW graph (Hierarchical Navigable Small World) instead of a brute-force scan: roughly O(log N) per query at ~0.99 recall@10 with the built-in parameters M=16 out-edges per node on the upper layers (32 on layer 0) and a query beam width of efSearch=64 (automatically raised to k when k is larger). Every vector is L2-normalized on insert, so similarity is a plain dot product. Two implementation details matter operationally: a node's graph level is derived deterministically from its id (hash-based, not an RNG), so replicas that insert the same ids build structurally similar graphs regardless of insert order or restarts; and deletes tombstone the graph node — it remains a routing waypoint but is filtered from results.

The text-protocol commands operate on the built-in vector namespace "default", whose dimensionality is fixed by the first VSET:

$ nc localhost 9000
VSET doc-1 0.1 0.9 0.2
OK
VSET doc-2 0.8 0.1 0.1
OK
VSEARCH 2 0.15 0.85 0.2
doc-1 0.997875
doc-2 0.312625
END

Persistence and replication: the HNSW graph itself lives in RAM only. Each vector is additionally persisted through the ordinary PUT path as a little-endian float32 blob under the reserved key @vec/<ns>/<id>, so vectors survive restarts: on startup (after WAL replay finishes) the engine walks the @vec/ keyspace and rebuilds every index, auto-registering each namespace with the dimension inferred from the blob length. In cluster modes the persisted blob replicates as a plain KV write, and the replica's apply hook detects the @vec/ prefix and refreshes its own in-RAM index — so VSEARCH on a replica reflects vectors written elsewhere.

Atomic operations

VeltrixDB provides four single-key atomic primitives: CAS (compare-and-swap), INCR / DECR (integer increment/decrement), and SETNX (set-if-not-exists). All four are implemented as a shard-locked read-modify-write: the operation holds the write lock on the key's index shard for the full duration of the read, compare, and durable persist (WAL + VLog fdatasync), so no interleaved write from another client can observe or clobber an in-flight atomic operation. Because the index is split across shards, a lock held during one key's atomic RMW only blocks the other keys that happen to hash to the same shard — not the whole keyspace.

Safe under concurrency: because the shard lock spans the entire compare-then-write sequence, two concurrent CAS calls (or an INCR racing a CAS) on the same key cannot interleave — one completes fully before the other's read is even taken. The trade-off is that the shard is briefly unreadable for other keys on that shard during the group-commit window (typically single-digit milliseconds); this is an intentional correctness-over-latency choice for atomic ops specifically, not for plain PUT/GET.

Transactions

VeltrixDB exposes optimistic, multi-key transactions through a Txn builder: stage a set of writes and deletes, optionally guard individual writes with an expected version, then call Commit.

Each SetIf guard captures the version of the key it expects to overwrite. At Commit time, VeltrixDB re-checks every guarded key under its shard lock; if any key was written by someone else in the meantime, the whole transaction aborts with ErrTxnConflict and nothing is applied — the caller is expected to re-read and retry. This is optimistic concurrency control: conflicts are detected at commit time rather than prevented up front with locks, so transactions never block each other while staging.

Concretely, this gives you conflict detection, not automatic conflict resolution — VeltrixDB tells you a concurrent writer got there first and lets your application decide how to retry or merge, rather than silently picking a winner. On the write path, VeltrixDB's own within-cluster replication engine additionally tracks per-replica vector clocks to establish causal ("happened-before") ordering between writes seen by different replicas during anti-entropy — the same detect-don't-resolve philosophy applied to replica reconciliation rather than to a single client's transaction.

Within a single disk, a transaction's batched writes are durably applied as one atomic unit (all or nothing); across disks, a transaction is not strictly serializable — each disk independently guarantees its own batch is atomic, but there is no two-phase commit tying the disks together. Read isolation is read-committed, not snapshot isolation: a Txn.Read always sees the latest committed state, never a stale snapshot.