Admission Control & GC

VeltrixDB continuously measures read latency and uses it as a feedback signal to throttle writes and pace VLog garbage collection — protecting foreground reads from being starved by background write and reclaim traffic.

Why admission control exists

NVMe bandwidth and IOPS are shared between three consumers: foreground reads, foreground writes, and background VLog GC. Without coordination, a burst of write traffic (or a GC pass reclaiming a large garbage backlog) can saturate the disk queue and push read tail latency into double-digit milliseconds. VeltrixDB's admission control closes this loop automatically: it watches read latency and, when it degrades, throttles the traffic that is competing with reads — rather than requiring an operator to manually cap write rate or pause GC during an incident.

The read-latency EWMA

Every Get() call feeds an exponentially-weighted moving average of read latency (ReadLatencyEWMANs). This single signal drives every decision described below.

To keep the EWMA update itself from becoming a bottleneck at multi-million-ops/sec read throughput, the CAS-loop update is sampled 1-in-64 reads rather than run on every read — at 2M reads/sec this adds at most ~32 microseconds of lag to the signal, far below the millisecond-scale thresholds it drives.

The admission-control ladder

As read latency rises, VeltrixDB escalates through a small set of protective actions:

Two things are worth calling out about this ladder:

• The GC bandwidth cap fires before the full write throttle. At 3 ms EWMA, GC is capped to 60 MB/s — a soft signal that gives GC a chance to back off before things get bad enough to touch client-visible write latency. Only at 4 ms does the system start actively slowing down PUT calls.
• The resume threshold (2 ms) is lower than the trigger threshold (4 ms). This hysteresis gap prevents the system from flapping in and out of throttled mode when latency is oscillating right around a single threshold.

The staleness guard

If no Get() call arrives for 4 minutes, the EWMA is treated as stale and GC is unconditionally resumed. Without this guard, a single slow read early in a write-only workload (e.g. a bulk load or benchmark with no read traffic) could latch the EWMA above the throttle threshold and never clear it — permanently blocking VLog GC because there are no subsequent fast reads to bring the average back down. The staleness window is tracked via LastReadNs, stamped on every Get().

Interaction with VLog GC tiers

The read-EWMA ladder above and the garbage-ratio-driven GC tiers (see Storage Engine: VLog GC tiers) are two independent signals that combine at GC decision time:

In plain terms: as long as garbage is below 65%, a paused-by-admission-control state (read EWMA ≥ 4 ms) is honored — GC stands down and gives reads the full disk. Only once garbage crosses the 65% emergency threshold does GC override that pause. This ordering is deliberate: below 65%, protecting read latency is the priority; above 65%, the system is already in enough trouble that continuing to defer GC would make things categorically worse.

The emergency tier: breaking the death spiral

The scenario the emergency tier exists to prevent: read latency rises → admission control pauses GC → garbage keeps accumulating because nothing reclaims it → VLog grows → cache hit rate drops (more cold data to page through) → read latency rises further → GC stays paused indefinitely. This is a genuine death spiral — without an escape hatch, a transient latency spike can turn into a permanent, self-reinforcing degradation.

At ≥ 65% garbage ratio, VeltrixDB breaks the cycle unconditionally: GCPaused is bypassed, GC bandwidth is uncapped, and the defrag interval is quartered so the system checks in on the disk far more frequently. Every emergency pass logs loudly:

[gc] disk=N EMERGENCY  garbage=67.2%  ewma=21.4ms  bypassing admission-control pause to escape death spiral

and increments veltrixdb_vlog_gc_emergency_runs_total.

Operational guidance

Metrics to watch

Healthy pattern

In a well-provisioned cluster, the garbage ratio oscillates within the 30–50% band, GC bandwidth stays at the normal cap, the write-admission-throttle counter is flat at zero (or increments only briefly during isolated traffic spikes and then stops), and the emergency-runs counter never moves. Read EWMA stays comfortably under the 3 ms first-tier threshold under normal load.

Unhealthy pattern

Watch for these signals together, since any one in isolation can be a false alarm but the combination is not:

• veltrixdb_vlog_garbage_ratio climbing steadily past 50% and staying there rather than oscillating back down after a GC pass.
• veltrixdb_storage_write_admission_throttles_total climbing at a sustained, non-trivial rate — this means read latency is high enough to be actively slowing down writes, not just an isolated blip.
• veltrixdb_vlog_gc_emergency_runs_total incrementing repeatedly rather than a single one-off spike.

This combination means write rate has outpaced GC reclaim capacity on one or more disks for a sustained period. The fix is capacity, not configuration: add NVMe IOPS/bandwidth headroom, reduce write rate to the affected disks, or check for a hot-shard skew that is concentrating write traffic on a single disk (see the shard-to-disk routing model in Storage Engine).