feat(remote): health monitor probe and status transitions (S13 Step 14, part 3/3)

David Fernandezrequested to merge
10io/s13-step-14-part3of3 into main

📦 What this MR does

Part 3 of 3 of S13 Step 14 (health monitoring): the HealthMonitor itself — the probe operation, the status transitions, and the audit trail, over part 1's held client config and part 2's counter seam and fakes.

  • Probe issues a HEAD through the step 2 seams (RequestBuilder + UpstreamDoer, under the fail-small BodyKindMetadata cap) with the probe_timeout deadline on a derived context, reads the prior status, advances it, writes the durable last_health_status/last_health_checked_at as a compare-and-set on the probe timestamp that returns the status it replaced — the transition is judged on that, never on the pre-probe read, so interleaved probes cannot drop a recovery event, and an overtaken probe writes nothing and audits nothing. The recording phase runs on a context detached from the caller so an exhausted caller budget cannot swallow the streak update and the write, and Recorded tells a caller whether anything persisted, and audits every transition plus every manual probe (From == To when nothing changed — the operator-ratified no-new-field shape).
  • Reachability is open on the 4xx side: any answered status below 500 is reachable, and a probe failure is a transport error, a timeout, or a 5xx — the spec's unhealthy row. An answer proves the upstream is up, so a 405 on a HEAD to the base URL (common on CDN-fronted mirrors) no longer takes an upstream with a working GET path out of resolution permanently. The taxonomy decision is being recorded on work item 320 alongside the 304 mapping.
  • Transitions follow the consecutive-failures contract: unknown holds below unhealthy_threshold, the threshold flips unhealthy, one reachable probe restores healthy and resets the streak. Counter errors degrade instead of failing the probe: an uncountable streak never flips a remote unhealthy (WARN-logged, status held), and a reachable upstream sets healthy no matter what the counter says.
  • Counter errors never fail the probe and never flip a status: an uncountable streak holds the current one, and such a degraded probe writes nothing durable at all — last_health_status and last_health_checked_at record conclusive probes only, so a held status cannot read as freshly confirmed. A counter call that fails with a context error logs at DEBUG, so shutdown raises no false cache alarm. One window is accepted and documented on the type: a Reset that fails during a cache outage leaves a stale streak (bounded by the counter TTL), so one failure after recovery can flip unhealthy early, worth at most one sweep interval and healed by the next passing probe. The complete fix is the cache-side atomic reset-and-count in #443; the process-local mark that tried to close it was removed after two demonstrated same-remote races.
  • Caller abandonment is cancellation only, the rule Prober.Probe follows: a canceled caller gets the abandoned error — wrapping both the context error and the transport or build error, so a sweep's shutdown check can match either with errors.Is — while any deadline expiring, the caller's own or the derived per-probe one, is an upstream that did not answer and counts as a failure.
  • Wiring and policy are guarded at the edges: a target with no Builder, or a monitor with no doer, returns ErrNotImplemented before any read or write (one unresolvable remote must not panic the sweep goroutine), and NewHealthMonitor rejects a non-positive ProbeTimeout or UnhealthyThreshold the way NewFailureCounter rejects the field it owns.

Plan drift, called out per the drift carve-out: the tests exercise the canonical remotetest fakes rather than the plan's httptest sketch — the step 9 precedent; no concrete adapter exists until step 11.

⚙️ Why stacked

Step 14's full diff is ~1.8k reviewable LoC, so it ships as 3 stacked MRs, reviewed and merged bottom-up. This part is the top: parts 1 (composition-root wiring) and 2 (failure counter + fakes) are below. Its ~1k reviewable LoC is a dependency-forced indivisible unit — nearly every subtest exercises the one Probe entry point, so the tests-with-code rule forbids cutting the suite from the type.

🔗 References

  • Plan: docs/plans/2026-07-16-s13-virtual-remote-foundation.md — Step 14
  • Spec: docs/specs/S13-virtual-remote-foundation.md — health monitoring: probe, status transitions, failure counter, audit events

🎯 Spec coverage

Acceptance / contract point Test
Reachable set (any answered status below 500; boundaries pinned at 200 and 499) → healthy TestHealthMonitor_Probe_ReachableStatuses
5xx fails (boundary pinned at 500) TestHealthMonitor_Probe_UnreachableStatuses
Overtaken compare-and-set write audits nothing TestHealthMonitor_Probe_StaleWriteSkipsAudit
Transition judged on the status the write replaced TestHealthMonitor_Probe_AuditsAgainstTheReplacedStatus
Recorded reports persistence for applied, overtaken, and degraded probes TestHealthMonitor_Probe_RecordedReportsPersistence
unknown holds below threshold TestHealthMonitor_Probe_UnknownHoldsBelowThreshold
Transport failure is a failed probe (status 0) TestHealthMonitor_Probe_TransportFailureIsFailedProbe
Threshold flips unhealthy TestHealthMonitor_Probe_ThresholdFlipsUnhealthy
Success resets streak, restores healthy TestHealthMonitor_Probe_SuccessResetsStreakAndRestoresHealthy
Manual probe always audits, From == To on no change TestHealthMonitor_Probe_ManualTriggerAlwaysAudits
Counter outage degrades, never flips, never writes; accepted stale-streak window TestHealthMonitor_Probe_CounterOutageDegrades
Cache outage warns, shutdown stays at debug TestHealthMonitor_Probe_CounterErrorLogLevel
Repeat unhealthy scheduled probe stays audit-silent TestHealthMonitor_Probe_RepeatUnhealthyStaysSilent
Build/store/audit seam errors propagate TestHealthMonitor_Probe_SeamErrorsPropagate
Canceled caller → abandoned error; any expired deadline → counted failure TestHealthMonitor_Probe_CallerAbandonmentPropagates
Unwired builder or doer → ErrNotImplemented, nothing read or written TestHealthMonitor_Probe_UnwiredDependenciesAreNotImplemented
Non-positive ProbeTimeout or UnhealthyThreshold rejected at construction TestNewHealthMonitor_RejectsNonPositiveConfig
HEAD-only, BodyKindMetadata, derived probe_timeout deadline TestHealthMonitor_Probe_RequestDiscipline
Health path passes through to the builder TestHealthMonitor_Probe_HealthPathPassesThrough
Trigger names (scheduled/manual) TestHealthProbeTrigger_String

🔬 e2e scenarios

No scenario added or affected: the monitor is format-agnostic service-layer plumbing with no reachable request path until steps 11/12 wire resolution and step 15 schedules the sweep and the on-demand entry.

Related to #335 (closed)

📚 Stacked MRs (review/merge bottom-up)

Edited by David Fernandez

Merge request reports