feat(gauntlet): fault-injection harness that asserts invariants

David Burkerequested to merge
feat/gauntlet into main

Summary

A fault-injection harness that runs the sample app against real services and asserts invariants. It complements the unit suite rather than competing with it:

Unit tests Gauntlet
Answers "does it do what we intended?" "what did we not think of?"
Finds known regressions unknown unknowns
Runtime seconds tens of seconds per scenario
Where every MR nightly / on demand

Independent of !29 (merged) and !27 (merged) — this branch is off main and changes no library code.

AI disclosure: Claude Code — built the harness, mined this project's history for scenarios, and drafted this description.

Why, concretely: chaos.py ran the right scenario and still went green

benchmarks/chaos.py already killed workers and counted completions. #10 (closed) shipped through it. Measured against unfixed main:

All 3000 tasks completed successfully!
No duplicate tasks found.
chaos.py EXIT: 0          <- reports success
DLQ:        801 entries    <- every one a false "Exceeded max_rescues (0)"

That is a baseline run with no fault injection at all. Three reasons it couldn't see it, each now a design rule in the harness:

  1. The oracle only counted completions. In #10 (closed) the task completes — the sweep dead-letters it and the worker finishes it anyway — so nothing ever looked at the DLQ. The invariants are the product; the chaos is only the input generator.
  2. Runs ended before the periodic machinery engaged. A default run finishes in seconds; VTASKS_RESCUE_INTERVAL is 30s, so not one sweep fired. Compress the intervals and feed at a steady rate so faults land on live traffic.
  3. --id chaos-worker-N bypassed the liveness machinery entirely. alive:chaos-worker-0 never matches the alive:{hostname}-* scan and processing:chaos-worker-0 never matches the sibling pattern, so the alive set was always empty and sibling rescue never ran. It exercised only the unconditional own-key drain — the path that is the bug. Don't let harness setup disable the mechanism under test.

Its stall detector was also inert: if completed_tasks > 0: last_progress_time = current_time refreshes the timer whenever any task has ever completed, so once one lands the 30s timeout can never fire. That is the #6 (closed)/#7 (closed)/#8 class it was meant to catch.

Verdicts are three-valued on purpose

Some of what this finds is a real limitation whose fix belongs to a future design (#11). A harness that reports those red every night gets switched off within a week. So:

  • FAIL — an invariant we expect to hold did not. Evidence names the specific task ids or keys; "FAIL: conservation" alone is unactionable.
  • known — failed exactly as the scenario documents. Not a regression.
  • FIXED — a documented limitation stopped failing. Update the catalogue.
  • VACUOUS — the scenario never exercised its own fault. Exits non-zero. This is the explicit guard against the trap above: a pass from a run that finished before its faults fired is not evidence.

Testing

The acceptance test for a harness is whether it detects a known bug, so:

  • baseline fails on unfixed main — 17–23 spurious DLQ entries with the exact Exceeded max_rescues (0) / _rescue_count: 1 signature, task ids listed — and passes with the !29 (merged) fix applied. Red-then-green against a real bug.
  • dlq_records_real_failures reports 5/5, which keeps every other DLQ assertion from being vacuous: an empty DLQ is only meaningful if deliberately-failing tasks do arrive in it.
  • Verified the vacuity guard by hitting it for real: the first version of driver-goes-quiet finished in 2.0s against a fault scheduled at t+8s and reported pass. That is what motivated the feeder and the VACUOUS verdict.

Not yet verified: the broker-* and db-latency scenarios depend on netfault.sh, which I have not confirmed actually changes measured behaviour (my first attempt measured over loopback, which netem on eth0 never touches). Marked unverified in scenarios.py and the README — an injector that silently does nothing is the vacuous-pass trap in a different costume.

Design notes worth reviewing

  • Ledger is a directory of files, not ChaosLog. ChaosLog writes to Postgres, so it can't record anything while the DB is the fault target, and it shares a datastore with the queue when the DB backend is under test. O_EXCL makes duplicate detection free — a second execution loses the race rather than needing a later dedupe pass.
  • Per-run Valkey keyspace {vtg-<run_id>}. Without it, processing: debris from an earlier run under a different container hostname is indistinguishable from a leak in this one. Diagnosing that false positive is how the cross-hostname leak turned up as a concrete artefact: because both rescue scans are hostname-scoped, that debris is never reclaimed by anything.
  • Quiescent assertions with a bounded poll, never a fixed sleep. A SIGTERMed worker drains before exiting, so asserting too early reports the drain in progress as leaked state.
  • Three distinct "blocks forever" shapes, because they differ in detectability: blocking the event loop stops the heartbeat too (an operator can see it), while awaiting forever or blocking a to_thread lane leaves a perfectly healthy-looking process with zero throughput. vtasks has no per-task timeout, so all three are unbounded; declared as known limitations pointing at #11.

Roadmap in the README

The scenario roadmap comes from mining this project's own history and issues rather than from imagination — each entry names a mechanism found by inspection. The highest-value unbuilt ones include a heterogeneous fleet with delayed tasks (_promote_loop takes one global lock then promotes only self.queues, so run_after tasks on another worker's queues are never promoted, with no error anywhere), a hostname change on restart, two workers sharing a worker_id, and a heartbeat lapse under event-loop load.

Merge request reports