feat(remote): scheduled health sweep as the first periodic job (S13 Step 15)
🩺 What this MR does
Implements S13 Step 15 (health monitoring — scheduled sweep + wiring): the first S27-A periodic River job, kind remote:health-sweep, which probes every remote repository once per health_check.scheduled_interval through the Step 14 HealthMonitor, refreshing durable health status even for remotes no request path touches.
internal/remote/healthjob.go— the sweep worker. Per-format sealedHealthSweepSource{Remotes, Targets, Prober}triples keep enumerator, target resolver, and prober from being mixed across formats. The sweep shuffles the fleet each run (jitter without a config knob) and paces probes evenly across the interval window with an in-flight cap of 10.RegisterPeriodicJobs(rc, deps)— establishes the per-package registration convention inside thewire_jobs.gobuild closure. The composition root reaches it through aperiodicRegistration{kind, register}entry each tier appends onto thewiring(wireRemotecontributes this one), sowireJobstakes one collected slice and grows no parameter, call-site argument, or feature-package import per future consumer;kindis the test handle that fails when an append is dropped. The composition root registers with an empty source aggregate: registration, scheduling, and worker are live, but the sweep is a no-op until the remote slices (S14/S15/S16) contribute their enumerators — the end-to-end sweep over real rows is a remote-slice acceptance concern per the plan.internal/config/virtualrepositories.go— boundsscheduled_intervalto (0, 1h): the worker'sTimeout()equals the interval, and keeping it under River's one-hour stuck-job rescue horizon means the worker's own budget always ends a run first — so the bound never leans on how the rescuer treats a job that is still working, and an orphaned sweep from a dead pod is still recovered within about an hour.internal/remote/remotetest—FakeHealthProberandFakeHealthTargetResolverfor the sweep suite and future remote-slice tests.
⚙️ Behavior contracts
Workalways returnsnil. A returned error makes River retry with ~1s backoff, so a deploy-time cancellation would trigger a duplicate fleet sweep; incomplete coverage is logged (INFO on shutdown cancellation, WARN otherwise) instead of signaled as a retryable failure.- The pacer stops releasing probes once
time.Until(deadline) < probe_timeout.HealthMonitor.Probedeliberately counts a caller deadline as an upstream failure (the S17 on-demand contract), so a probe cut by the job deadline would durably increment a healthy remote's failure streak. - Probe goroutines carry their own
recover(): River only guards theWorkgoroutine.docs/dev/background-jobs.mddocuments the scoping. RunOnStartis false — a fleet-wide probe burst on every deploy is the exact thundering herd the pacing exists to avoid.HealthMonitor.Proberefuses aHealthProbeTarget.HealthPathcarrying a scheme or an authority component withErrHealthPathNotRelative, before any request build, send, prior-status read, durable write, or streak advance. The field's constraint — the path reaches the request builder verbatim, so an authority component resolves the probe against a host the path chose — was carried by its doc comment alone; the check now holds it for every resolver, the sweep's and S17's connection test alike.
📈 Scalability follow-up
With the in-flight cap of 10 and the default 5m interval / 5s probe timeout, one sweep covers ~590 remotes worst-case (every probe at timeout) and ~15k typical. Raising that ceiling is deliberately out of scope here and tracked in #479:
- probe dedupe by
(format, URL, healthPath)— many remotes pointing at the same public upstream (for example Maven Central) collapse to one HEAD with the outcome fanned out per row; - pagination/streaming for the remote-repository enumerators;
- a per-sweep released/total coverage metric.
✅ Testing
internal/remote/healthjob_test.go— unit suite against fake sources: coverage across sources, error and panic isolation per probe, the deadline release gate, cancellation without retry signal, the in-flight cap, pacing, and shuffle.internal/remote/healthjob_integration_test.go— realjobsriver.Clientover the testcontainers database:RegisterPeriodicJobsalone makesStartsucceed and a leader-elected scheduled fire probes the fleet end to end.internal/config/virtualrepositories_test.go— the newscheduled_intervalbounds.
🎯 Spec coverage
Plan: docs/plans/2026-07-16-s13-virtual-remote-foundation.md — Step 15. Acceptance sentences and the behavior contracts above, mapped to the tests that pin them.
| Acceptance / contract point | Test |
|---|---|
The job registers and Start succeeds on the registration alone |
TestRegisterPeriodicJobs_Integration_SweepFiresOnSchedule |
The leader-elected enqueuer schedules the sweep within one scheduled_interval, and the delivered job probes the fleet |
TestRegisterPeriodicJobs_Integration_SweepFiresOnSchedule |
| Durable routing kind never drifts | TestHealthSweepArgs_Kind |
Timeout overrides River's one-minute default with scheduled_interval |
TestHealthSweepWorker_TimeoutIsScheduledInterval |
| Per-remote jitter: every remote lands at a random phase | TestHealthSweepWorker_ShufflesProbeOrderAcrossSweeps |
| Starts paced evenly across the interval window | TestHealthSweepWorker_PacesProbesAcrossTheInterval |
| Every enumerated remote of every source is probed through its own seams, with the scheduled trigger | TestHealthSweepWorker_ProbesEveryRemoteAcrossSources |
Durable columns updated for each iterated entry, composed with the real HealthMonitor |
TestHealthSweepWorker_ComposedWithMonitorWritesDurableColumns |
| At-least-once contract: a duplicate fire costs one extra increment per remote, flips nothing below the threshold, and clears on any success | TestHealthSweepWorker_DuplicateFireDistortsStreakByOne |
| Empty source aggregate is a live registration and a no-op sweep | TestHealthSweepWorker_EmptyAggregateIsNoOp |
| A probe failure never ends the sweep | TestHealthSweepWorker_ContinuesPastProbeErrors |
| A target-resolution failure skips one remote | TestHealthSweepWorker_SkipsRemoteOnTargetResolutionError |
| An enumeration failure skips one source, not the sweep | TestHealthSweepWorker_SkipsSourceOnEnumerationError |
| A probe panic is recovered per remote, not per process | TestHealthSweepWorker_RecoversProbePanics |
In-flight probes capped at healthSweepProbeConcurrency |
TestHealthSweepWorker_CapsInFlightProbes |
One ProbeTimeout reserved at the deadline, so no probe is cut mid-flight |
TestHealthSweepWorker_ReservesProbeBudgetAtDeadline |
| Cancellation ends the sweep with no retry signal | TestHealthSweepWorker_CancellationAbortsWithoutRetrySignal |
scheduled_interval bounded to (0, 1h), with 59m accepted |
TestLoad_VirtualRepositories_OutOfRangeKnobs, TestLoad_VirtualRepositories_ScheduledIntervalBelowRescueWindowAccepted |
HealthPath must be relative: a scheme or authority is refused before any build, send, read, write, or streak advance |
TestHealthMonitor_Probe_RejectsNonRelativeHealthPath, TestHealthMonitor_Probe_AcceptsRelativeHealthPath |
🎯 E2E scenarios
No scenario catalog update: the sweep has no user-facing surface, and the source aggregate is empty until a remote slice contributes its enumerator — the observable end-to-end behavior lands with those slices and their catalogs.
Related to #336 (closed)