Benchmarking Methodology

How VeltrixDB's benchmark numbers are produced: the harness, the hardware, the pass/fail gates, and the raw results with the conditions they were measured under — so you can reproduce them or run the same harness against your own workload shape.

The harness

scripts/bench.sh runs a 6-phase sequenced workload (build + start, MPUT bulk load, cool-down, read-only, mixed 70R/30W, sustained MPUT write stress) and reports go/no-go on two hard gates: packing density and GC emergency runs. The project re-runs it after any storage engine change.

It builds the server and loadtest with CGO_ENABLED=0 (Go map index) unless you export CGO_ENABLED=1. The Docker image is a cgo build (native index), so set CGO_ENABLED=1 on Linux to measure what the image runs.

# macOS / Linux dev (uses /tmp, 1M keys)
./scripts/bench.sh

# Linux production-class (n2-highmem-64, 8 NVMe disks)
DATA_DIRS=/mnt/nvme0,...,/mnt/nvme7 \
RAW_VLOGS=/dev/nvme0n1,...,/dev/nvme7n1 \
CACHE_MB=409600 NUM_KEYS=10000000 \
CONCURRENCY=512 BULK_DUR=120 STRESS_DUR=300 \
  ./scripts/bench.sh

Key environment variables

Pass criteria

The harness enforces two hard gates. A benchmark run that fails either gate is not representative of the storage engine operating correctly.

The script exits non-zero if either gate fails. It also prints the read P99, but that is not a gate.

Reference numbers (BENCHMARKING.md)

macOS M-series (dev)

Linux n2-highmem-64 (8 NVMe) — projected

YCSB benchmark results (100M keys)

These are the only reproducible headline numbers. Published in BENCHMARK_RESULTS.md, dated June 9, 2026, on a historical pure-Go build (v1.0; it predates the native index, the binary WAL, the one-write(2) WAL flusher, the ordered-index fix and --net). Test environment: AWS EC2 high-memory instance, 495 GB total RAM (488 GB available), 4× NVMe SSD at 873 GB each (3.5 TB total), raw block-device VLog (/dev/nvmeXn1p2), 400 GB LIRS cache, 200 client threads, 100,000,000 keys, ~100-byte values, benchmark tool YCSB 0.17.0. The YCSB binding used for this run is not in the repository, so it cannot be re-run from it; bench/compare (go-ycsb, below) is the reproducible YCSB path.

Write benchmark (100M key load)

Configuration: 5 ms WAL flush window, 5 ms VLog flush window (the server default is now 15 ms), 200 threads, 4× NVMe striped, every write fdatasync-durable.

Read benchmark (10M reads on 100M key dataset)

Configuration: 400 GB cache, 200 threads, 10,000,000 operations, 100% reads (YCSB workload_c). The entire dataset was in cache, so every read was a cache hit — this measures the cache-hit path through the network stack, not NVMe reads.

Progression across runs

BENCHMARK_RESULTS.md tracks how throughput and hit rate changed as the dataset and tuning scaled up:

Later measurements New in v1.5–v1.7

Not YCSB and not the YCSB machine; each row states its conditions. These cover the native index and binary WAL (v1.6) and the ordered-index fix and C++ front-end (v1.7).

Because the CI rows use tmpfs and a client on the same host, compare them with each other, not with other hardware. Reproduce them with scripts/net-bench.sh or the “Net front-end” workflow: ENGINES=nocgo NETS=go WINDOWS="5 1" for the “C++ storage off” rows, ENGINES=sqpoll for the “all on” row, NETS="go uring poll" for the front-end row. The macOS 3.54M row is the same batch phase (loadtest --mode write --concurrency 8 --batch-size 1024 --num-keys 1000000) against a local server.

Network front-ends and storage configurations (net-bench.sh) New in v1.7

scripts/net-bench.sh starts a fresh server per combination and runs batch write (8 × 1024-key MPUT, BATCH_DURATION, default 10 s), read (64 binary GET clients) and mixed (70/30) workloads, then prints one markdown table (also to $GITHUB_STEP_SUMMARY). The “Net front-end” GitHub workflow runs it on Linux, where the io_uring code exists.

# Front-ends (NET:N = N event loops for a C++ front-end)
NETS="go uring poll uring:2" ENGINES=native WINDOWS=5 ./scripts/net-bench.sh

# Storage configurations under the Go front-end
ENGINES="native bridge sqpoll map nocgo" NETS=go WINDOWS="5 1" ./scripts/net-bench.sh

Defaults: NETS="go uring", ENGINES=native, WINDOWS=5, DURATION=15 s per read/mixed workload, NET_THREADS = CPU count. The C++ front-ends need the binary protocol, so reads and mixed run with loadtest --proto=binary.

• Every overwrite appends to the WAL and VLog and nothing reclaims space during a run; one combination leaves ~5–10 GB, so each data dir is deleted when its combination finishes.
• Each workload runs under a watchdog (duration + WATCHDOG_SLACK, default 60 s). On a hang the script dumps native and Go stacks, records a FAILED row, and moves on.
• C++ front-ends log per-stage latency at shutdown (log2 buckets, so p99<=8ms means [4, 8) ms). A high cb_enter means loop threads are waiting for a Go P; try fewer loops (uring:2).
• On macOS only poll with one loop runs, and loopback caps round trips at ~60K/s for both front-ends — measure on Linux.

October 2026 measurements New in v1.11–v1.12

Taken while building adaptive group commit, vector / text / hybrid search, PQ and the disk graph. All on one macOS laptop (Apple silicon) unless stated, so absolute numbers do not predict a Linux NVMe server, and none of them is a comparison with another database. Each table names the test or tool that reproduces it. Source: BENCHMARK_RESULTS.md, “October 2026 measurements”.

Durable single-key writes: adaptive vs fixed group commit

--group-commit=adaptive (the default since v1.12) vs fixed. The device’s fdatasync cost is emulated (a sleep before the real sync), because macOS fsync returns at the drive cache; that is what makes the result mean the same on a laptop as on Linux. 15 ms window, 128 B values, each writer doing sequential durable Puts for 1.5 s.

VELTRIX_GC_TABLE=1 go test ./storage -run TestGroupCommit_LatencyTable -v

Emulated sync = 300 µs (NVMe-like):

Emulated sync = 2 ms (slow / network disk):

TestGroupCommit_AdaptiveGates (every PR) fails if a lone writer waits for the window or concurrent writers stop sharing syncs. The nightly workflow prints the table in its job summary.

YCSB A–F through the server (go-ycsb, VeltrixDB only)

bench/compare/cmd/ycsb (go-ycsb core workloads) against a local server: 20,000 records × 10 fields × 100 B, 20,000 operations, 16 threads, zipfian, whole-record writes. Client and server on the same laptop; macOS fsync does not reach the platter, so write latency here is a lower bound. Not comparable with the June YCSB 0.17.0 run.

cd bench/compare && START=0 DBS=veltrixdb RECORDS=20000 OPS=20000 THREADS=16 WORKLOADS="a b c d e f" ./compare.sh

compare.sh defaults: DBS="veltrixdb aerospike scylla", WORKLOADS="a b c f", RECORDS=1000000, OPS=1000000, THREADS=64, START=1 (docker compose up / stop each database in turn). With START=0 it uses databases already running at VELTRIX_ADDR (default 127.0.0.1:9000), AEROSPIKE_HOST and SCYLLA_HOSTS. bench/compare is a separate Go module (Go ≥ 1.25).

Vector search

Real embeddings. First 100,000 words of GloVe-100 (angular), 100 dimensions, 500 queries, exact cosine ground truth. In-process, one query at a time, macOS (Apple silicon, GOMAXPROCS = 18). Build = 18 parallel durable PutVector writers.

scripts/ann-dataset.py glove-100-angular /tmp/ann --train 100000 --test 500
VELTRIX_ANN_DIR=/tmp/ann go test ./storage -run TestRealEmbeddings -v -timeout 30m

The quantized layouts are slower per query than float32 at this size because they re-rank from the VLog (one parallel MultiGet; before that change int8 at ef = 64 was p50 1.9 ms / p99 13.8 ms). Their point is RAM: at 100 dimensions the vectors are small and graph edges dominate. scripts/ann-dataset.py needs h5py and numpy.

Over the network (vecbench). GloVe-100, first 20,000 words, 300 queries, int8, 8 concurrent clients, server and client on one macOS machine. Load: 6.4 s (3,146 durable vector writes/s).

cd bench/compare && go run ./cmd/vecbench -data /tmp/ann -train 20000 -quant int8 -threads 8
# flags: -addr (default 127.0.0.1:9000) -efs (default 64,128,256,512) -queries -pqm -graph memory|disk -out results/

RAM per vector, 768 dimensions. 10,000 synthetic clustered vectors, Go heap per vector, macOS.

VELTRIX_VECTOR_MEMORY=1 go test ./storage -run TestVectorMemoryTable -v

Tested up to 100K real vectors and 10K at 768 dimensions; nothing at 10M+ yet.

Search soak and crash chaos

The nightly workflow runs a 20-minute soak and 10 crash cycles on Linux:

VELTRIX_SOAK_DURATION=20m go test ./tests/integration/ -run TestSearchSoak -v -timeout 40m
VELTRIX_CHAOS_CYCLES=10 go test ./tests/integration/ -run TestSearchCrashRecovery -v -timeout 25m

Writes in flight at the kill are not checked. After each crash cycle the test rewrites every id whose write had an unknown outcome to a fresh, fully acknowledged version, so later cycles check those ids too.

Consistency and rolling-upgrade tests New on main after v1.16 — ships in the next release

Jepsen-style harness (tests/jepsen, Go standard library only, all in _test.go files): 3-node raft and replicated clusters of real server processes, 8 clients on the text protocol (GET, PUT, CAS, INCR, set adds), and a nemesis that SIGKILLs or SIGSTOPs the leader or a random node, crashes a follower mid-InstallSnapshot, and — through an iptables hook on Linux as root — partitions it. Checkers: per-key linearizability (Wing & Gong search with memoisation, shrunk to a minimal failing history), counters, sets, read validity and copy counts. A negative control must find stale reads.

go test ./tests/jepsen/ -short -timeout 120s          # checker unit tests only
go test ./tests/jepsen/ -v -timeout 15m               # every mode, 20 s of workload each
VELTRIX_JEPSEN_DURATION=5m VELTRIX_JEPSEN_SEED=42 \
  go test ./tests/jepsen/ -v -run TestJepsenRaftLinearizable -timeout 20m

Limits: one host (shared disk, kernel and clock; faults are process-level), no real partitions on macOS, no clock skew, power loss not simulated, single-key text-protocol operations only (no multi-key TXN, MPUT, namespaces, search or binary protocol).

Rolling-upgrade tests (tests/upgrade, VELTRIX_UPGRADE_TEST=1): real v1.16 and current-build processes on the same data dirs — raft rolling upgrades with and without snapshots, mixed-version snapshot transfers in both directions, a replicated rolling upgrade, standalone upgrade and downgrade (clean stop and SIGKILL), and raft downgrade with and without a wipe. They found the v1.16 snapshot TTL loss on upgrade (fixed) and corrected the downgrade guidance.

VELTRIX_UPGRADE_TEST=1 go test ./tests/upgrade/ -v -timeout 30m   # builds v1.16 from the tag

Both run nightly (.github/workflows/nightly.yml, jobs jepsen — with and without --raft-pipeline, partitions enabled — and upgrade).

Against Aerospike and ScyllaDB (same hardware)

bench/compare/compare.sh runs go-ycsb workloads through a VeltrixDB driver and go-ycsb’s Aerospike and Cassandra (ScyllaDB) drivers, one database at a time on the same machine, and writes results/<ts>/summary.md.

# On the machine under test (Linux, NVMe), with Docker and Go ≥ 1.25:
export BENCH_DATA=/mnt/nvme0/bench BENCH_CPUS=8 BENCH_MEM=16g
./compare.sh                                   # all three, workloads a b c f
RECORDS=10000000 OPS=10000000 THREADS=128 WORKLOADS="a b c d f" ./compare.sh

# Vectors (VeltrixDB only):
../../scripts/ann-dataset.py glove-100-angular /mnt/nvme0/ann --train 1000000 --test 1000
VECTORS=1 ANN_DIR=/mnt/nvme0/ann VEC_ARGS="-quant pq -graph disk" ./compare.sh

Publish a result only with all of these stated next to it: the same machine and disk with one database running at a time (a client on a second machine is better still); the durability setting of each (VeltrixDB ACKs after fdatasync of WAL and VLog; ScyllaDB’s default commitlog_sync: periodic ACKs before the write is on disk, so use batch or report both; Aerospike ACKs from its write buffer); warm-up and a dataset larger than the caches if the point is disk performance; the replication factor (1 with NODES=1, 3 with NODES=3); and the versions of all three.

3-node clusters, replication factor 3 Fixed in v1.15

NODES=3 sets RF 3 for every database and needs three addresses per database. For VeltrixDB: VELTRIX_ADDRS (comma-separated), VELTRIX_MODE=raft|replicated, and VELTRIX_RAFT_PIPELINE=true|false so the summary records which --raft-pipeline setting the servers ran. The VeltrixDB driver homes threads round-robin over the addresses and reads locally; writes follow the first MOVED <leader> and share the leader address across threads; MOVED - (election running) waits and retries; a connection error fails over to the next listed node. After the run, cmd/keycount checks that every node holds every record. The guarantees differ per database (raft: quorum commit through the leader; ScyllaDB: QUORUM; Aerospike CE: all three copies), so state them next to any number.

# servers already running on three hosts
START=0 NODES=3 VELTRIX_MODE=raft \
  VELTRIX_ADDRS=10.0.0.11:9000,10.0.0.12:9000,10.0.0.13:9000 DBS=veltrixdb ./compare.sh

# GCP: 3 server VMs with local NVMe + 1 client, raft-off / raft-on / replicated, then deletes everything
PROJECT=<your project> ./gcp-multihost.sh

START=1 NODES=3 runs three containers per database on one machine (docker-compose.cluster.yml) to check that clusters form and drivers route — not for performance. That compose file is syntax-checked only; the 3-node VeltrixDB path was run as local processes on one laptop, against raft and replicated clusters and with a raft leader killed mid-run. Laptop figures, which are not cluster results (three raft nodes share one drive): the incremental raft log took load from 70 to 1,046 inserts/s (one run each), and --raft-pipeline was slower on load (560 vs 1,038 inserts/s, medians of 2 runs). No multi-host run has been published yet.

Internal GKE run (unverified)

Reported conditions: 3-node GKE cluster · n2-highmem-64 · 8×375 GB NVMe per node · 1 billion keys · 80R/20W · 30 minutes sustained.

Single node, historical — mixed provenance

Pure-Go path with the fixed group-commit window (before adaptive group commit became the default in v1.12). Each row says whether it was measured, computed or projected; none is a current benchmark run.

VeltrixDB vs. Redis, head to head

The VeltrixDB column is from the YCSB run above. The Redis column was not measured — it is a range of commonly cited figures, included for orientation only.

VeltrixDB always writes durably. Redis requires appendfsync always for an equivalent guarantee, which reduces Redis to 10K–30K ops/sec — comparable to or slower than VeltrixDB at that durability level.

Key takeaways

• Writes: 18,064 durable writes/sec across 4 NVMe disks at a 5 ms window, zero failures across 100M operations.
• Reads: 427,697 reads/sec at 461 µs average latency, with the whole dataset in the 400 GB cache (100% hits). Cache-miss (NVMe) read performance was not measured in this run.
• Disk footprint: 100M keys used ~25 GB per disk out of 873 GB available. Larger key counts (1 billion+) have not been measured reproducibly; note the index itself stays in RAM (~142 B/key with the native index).
• Durability: every write is crash-safe (WAL + fdatasync). Redis needs appendfsync always for a comparable guarantee.
• Read hit rate: 100% with proper dataset sizing (recordcount matching operationcount), zero NOT_FOUND across 10 million reads.