feat(remote): scheduled health sweep as the first periodic job (S13 Step 15)

🩺 What this MR does

Implements S13 Step 15 (health monitoring — scheduled sweep + wiring): the first S27-A periodic River job, kind remote:health-sweep, which probes every remote repository once per health_check.scheduled_interval through the Step 14 HealthMonitor, refreshing durable health status even for remotes no request path touches.

  1. internal/remote/healthjob.go — the sweep worker. Per-format sealed HealthSweepSource{Remotes, Targets, Prober} triples keep enumerator, target resolver, and prober from being mixed across formats. The sweep shuffles the fleet each run (jitter without a config knob) and paces probes evenly across the interval window with an in-flight cap of 10.
  2. RegisterPeriodicJobs(rc, deps) — establishes the per-package registration convention inside the wire_jobs.go build closure. The composition root reaches it through a periodicRegistration{kind, register} entry each tier appends onto the wiring (wireRemote contributes this one), so wireJobs takes one collected slice and grows no parameter, call-site argument, or feature-package import per future consumer; kind is the test handle that fails when an append is dropped. The composition root registers with an empty source aggregate: registration, scheduling, and worker are live, but the sweep is a no-op until the remote slices (S14/S15/S16) contribute their enumerators — the end-to-end sweep over real rows is a remote-slice acceptance concern per the plan.
  3. internal/config/virtualrepositories.go — bounds scheduled_interval to (0, 1h): the worker's Timeout() equals the interval, and keeping it under River's one-hour stuck-job rescue horizon means the worker's own budget always ends a run first — so the bound never leans on how the rescuer treats a job that is still working, and an orphaned sweep from a dead pod is still recovered within about an hour.
  4. internal/remote/remotetestFakeHealthProber and FakeHealthTargetResolver for the sweep suite and future remote-slice tests.

⚙️ Behavior contracts

  • Work always returns nil. A returned error makes River retry with ~1s backoff, so a deploy-time cancellation would trigger a duplicate fleet sweep; incomplete coverage is logged (INFO on shutdown cancellation, WARN otherwise) instead of signaled as a retryable failure.
  • The pacer stops releasing probes once time.Until(deadline) < probe_timeout. HealthMonitor.Probe deliberately counts a caller deadline as an upstream failure (the S17 on-demand contract), so a probe cut by the job deadline would durably increment a healthy remote's failure streak.
  • Probe goroutines carry their own recover(): River only guards the Work goroutine. docs/dev/background-jobs.md documents the scoping.
  • RunOnStart is false — a fleet-wide probe burst on every deploy is the exact thundering herd the pacing exists to avoid.
  • HealthMonitor.Probe refuses a HealthProbeTarget.HealthPath carrying a scheme or an authority component with ErrHealthPathNotRelative, before any request build, send, prior-status read, durable write, or streak advance. The field's constraint — the path reaches the request builder verbatim, so an authority component resolves the probe against a host the path chose — was carried by its doc comment alone; the check now holds it for every resolver, the sweep's and S17's connection test alike.

📈 Scalability follow-up

With the in-flight cap of 10 and the default 5m interval / 5s probe timeout, one sweep covers ~590 remotes worst-case (every probe at timeout) and ~15k typical. Raising that ceiling is deliberately out of scope here and tracked in #479:

  • probe dedupe by (format, URL, healthPath) — many remotes pointing at the same public upstream (for example Maven Central) collapse to one HEAD with the outcome fanned out per row;
  • pagination/streaming for the remote-repository enumerators;
  • a per-sweep released/total coverage metric.

Testing

  • internal/remote/healthjob_test.go — unit suite against fake sources: coverage across sources, error and panic isolation per probe, the deadline release gate, cancellation without retry signal, the in-flight cap, pacing, and shuffle.
  • internal/remote/healthjob_integration_test.go — real jobsriver.Client over the testcontainers database: RegisterPeriodicJobs alone makes Start succeed and a leader-elected scheduled fire probes the fleet end to end.
  • internal/config/virtualrepositories_test.go — the new scheduled_interval bounds.

🎯 Spec coverage

Plan: docs/plans/2026-07-16-s13-virtual-remote-foundation.md — Step 15. Acceptance sentences and the behavior contracts above, mapped to the tests that pin them.

Acceptance / contract point Test
The job registers and Start succeeds on the registration alone TestRegisterPeriodicJobs_Integration_SweepFiresOnSchedule
The leader-elected enqueuer schedules the sweep within one scheduled_interval, and the delivered job probes the fleet TestRegisterPeriodicJobs_Integration_SweepFiresOnSchedule
Durable routing kind never drifts TestHealthSweepArgs_Kind
Timeout overrides River's one-minute default with scheduled_interval TestHealthSweepWorker_TimeoutIsScheduledInterval
Per-remote jitter: every remote lands at a random phase TestHealthSweepWorker_ShufflesProbeOrderAcrossSweeps
Starts paced evenly across the interval window TestHealthSweepWorker_PacesProbesAcrossTheInterval
Every enumerated remote of every source is probed through its own seams, with the scheduled trigger TestHealthSweepWorker_ProbesEveryRemoteAcrossSources
Durable columns updated for each iterated entry, composed with the real HealthMonitor TestHealthSweepWorker_ComposedWithMonitorWritesDurableColumns
At-least-once contract: a duplicate fire costs one extra increment per remote, flips nothing below the threshold, and clears on any success TestHealthSweepWorker_DuplicateFireDistortsStreakByOne
Empty source aggregate is a live registration and a no-op sweep TestHealthSweepWorker_EmptyAggregateIsNoOp
A probe failure never ends the sweep TestHealthSweepWorker_ContinuesPastProbeErrors
A target-resolution failure skips one remote TestHealthSweepWorker_SkipsRemoteOnTargetResolutionError
An enumeration failure skips one source, not the sweep TestHealthSweepWorker_SkipsSourceOnEnumerationError
A probe panic is recovered per remote, not per process TestHealthSweepWorker_RecoversProbePanics
In-flight probes capped at healthSweepProbeConcurrency TestHealthSweepWorker_CapsInFlightProbes
One ProbeTimeout reserved at the deadline, so no probe is cut mid-flight TestHealthSweepWorker_ReservesProbeBudgetAtDeadline
Cancellation ends the sweep with no retry signal TestHealthSweepWorker_CancellationAbortsWithoutRetrySignal
scheduled_interval bounded to (0, 1h), with 59m accepted TestLoad_VirtualRepositories_OutOfRangeKnobs, TestLoad_VirtualRepositories_ScheduledIntervalBelowRescueWindowAccepted
HealthPath must be relative: a scheme or authority is refused before any build, send, read, write, or streak advance TestHealthMonitor_Probe_RejectsNonRelativeHealthPath, TestHealthMonitor_Probe_AcceptsRelativeHealthPath

🎯 E2E scenarios

No scenario catalog update: the sweep has no user-facing surface, and the source aggregate is empty until a remote slice contributes its enumerator — the observable end-to-end behavior lands with those slices and their catalogs.

Related to #336 (closed)

Edited by David Fernandez

Merge request reports

Loading
Loading