feat(remote): upstream health failure counter (S13 Step 14, part 2/3)

David Fernandezrequested to merge
10io/s13-step-14-part2of3 into main

📦 What this MR does

Part 2 of 3 of S13 Step 14 (health monitoring): the consecutive-probe-failure counter and every test fake the health monitor (part 3) exercises.

  • FailureCounter is the health monitor's seam over the S05-A atomic counter, keyed by the remote repository's parent repositories.id UUID. The plan anticipated a forwarding adapter; none is needed — counter.Counter's method set matches the seam exactly, pinned by a compile-time assertion (plan drift, called out per the drift carve-out).
  • NewFailureCounter is the single production constructor: it bakes in the counter name (upstream_health_failures) and the sliding expiry (4 sweep intervals, per the S05-A sizing rule), so a wiring that reached for counter.New directly cannot drop the TTL and leave abandoned streak keys behind. It returns an error on a non-positive ScheduledInterval: that value sizes the whole expiry, and counter.WithTTL reads a non-positive duration as no expiry — the never-expiring counter the constructor exists to prevent. FailureCounterTTLMultiple is exported so the integration test bounds the observed expiry against the same constant the constructor applies.
  • The integration test proves the production value against real Redis through the cache-purpose client: streak accumulation, the ar:counter:<name>:<uuid> key grammar (a deliberate cross-package pin, the counter suite's own discipline), the baked-in sliding TTL, and reset-to-absent. It rides test:integration rather than the Redis topology matrix: the job already runs both backends, the single-key counter behavior is topology-covered by internal/cache/counter's own matrix suite, and this package's coming cache-fill integration tests need the PostgreSQL the topology jobs lack. internal/remote is listed recursively to match the job's recursive changes rule.
  • testutil.StartIsolatedRedisClient centralizes the Redis integration-test bootstrap (provision, construct through the caller's redisclient constructor, start, cleanup); this test is its first caller.
  • remotetest gains FakeFailureCounter (error injection leaves state untouched), FakeHealthAuditSink (the canonical capturing sink steps 15/S17 will reuse), and SetHealthStatus call recording on FakeHealthStatusStore, so part 3's probe tests can assert the durable write happens on every probe. The HealthAuditSink seam the fake stands in for ships here too, beside HealthTransitionEvent in audit.go, so the fake is pinned to it at compile time from the start; part 3 drops its own copy of the interface on its next rebase. The fakes live in remotetest rather than beside the counter — operator decision, matching the package's canonical-fakes convention.

⚙️ Why stacked

Step 14's full diff is ~1.8k reviewable LoC, so it ships as 3 stacked MRs, reviewed and merged bottom-up. Part 1 (composition-root wiring) is below this MR; part 3 (the health monitor consuming this seam and these fakes) sits above.

🔗 References

  • Plan: docs/plans/2026-07-16-s13-virtual-remote-foundation.md — Step 14
  • Spec: docs/specs/S13-virtual-remote-foundation.md — failure counter; S05-A for the counter contract and TTL sizing

🔬 e2e scenarios

No scenario added or affected: the counter is service-layer plumbing with no reachable request path until steps 11/12 wire resolution and step 15 schedules the sweep.

🚧 Follow-ups

  • Migrate the two pre-existing per-file Redis integration-test bootstraps onto testutil.StartIsolatedRedisClient (touching those files is outside this step's scope).

Related to #335 (closed)

📚 Stacked MRs (review/merge bottom-up)

Edited by David Fernandez

Merge request reports