feat(remote): report each probe to a format-scoped observer (S15 observability plan: 3/4)

Summary

Step 3 of the npm remote observability plan. remote.HealthMonitor gains an optional per-format probe observer and reports one observation per settled probe, carrying the repository identity, the probed host, the status, whether a request reached the transport, and the pass verdict. No format implements the hook yet: all six composition-root sites pass no observer, so behaviour on this branch is byte-identical to main and TestHealthMonitor_Probe_ObserverLeavesTheProbeUnchanged pins that.

The seam is shaped here and consumed by step 4, so this MR also settles the three things that are cheap now and expensive once npm reads it:

• The report is contained and ordered. observeProbe runs deferred, so the observation lands behind the durable write and the audit event on every return path, and it recovers a panicking observer. An unrecovered report ahead of the recording phase would have cost every remote of that format its advance, its SetHealthStatus and its RecordHealthTransition, so a dead upstream would never have flipped unhealthy.
• The report context is detached and bounded. context.WithoutCancel, matching the recording phase, under a healthObserveTimeout of its own. abandoned() matches only context.Canceled, so a caller whose deadline expired settles the probe and reports; an observer honouring its context would otherwise have dropped exactly the probes at the tail of a saturated sweep, which are the slowest upstreams. The budget opens inside observeProbe rather than where the report is deferred, so the recording phase's own five seconds cannot hand the report an already-expired context.
• A local refusal is distinguishable from an unreachable upstream. HealthProbeObservation.RequestSent is the discriminator, so step 4 branches on data rather than string-matching UpstreamHost against invalid-url. docs/dev/observability.md defines the read routes' invalid-url as a base url.Parse refuses, while this arm is any builder refusal, and the reachable one on npm is a stored token no header can carry.

Plan steps

This step's Depends on is none, and it is not stacked on another branch.

#	Step	This MR
1	the metadata routes observe their reads
2	retire the version field from npm_remote_proxy
3	HealthMonitor reports each probe to a format-scoped observer	this one
4	npm books the probe's upstream response and its record	needs this one merged

Accepted trade-offs and deliberate omissions

• NewHealthMonitor stays at seven positional parameters, as the plan specifies. A variadic WithProbeObserver would read better and matches this package's own FlowOptions.Observer idiom, but switching is a deviation from the merged plan's step 3 and belongs in a plan amendment plus its own refactor MR, not here.
• NewHealthMonitor's and Probe's doc comments do not mention the observer. Each is a single comment segment of 14 and roughly 40 lines, so under scripts/ci/check-comment-caps.sh touching either charges every line of it against the 3-line exported cap. The optionality is recorded instead in a body-head comment on the constructor and on HealthProbeObserver itself.
• RequestBuilder.BuildRequest's contract is unchanged. It promises a non-nil Header and never a non-nil URL, and its doc block has the same caps bind. issueProbe reads the host defensively instead, so a request carrying no URL settles as a failed probe rather than panicking, which is also what it did before this MR.
• The report's deadline is a signal, not enforcement. healthObserveTimeout bounds an observer that honours its context; one that ignores the deadline still holds the probing goroutine, and with it one of the sweep's healthSweepProbeConcurrency slots. Running the report in its own goroutine would enforce it, at the cost of the ordering the deferred call buys and of an unbounded goroutine per stuck observer. The bound plus an explicit contract on HealthProbeObserver is the proportionate answer while step 4 is the only implementer.
• The detached tail is now healthRecordTimeout + healthObserveTimeout. Seven seconds against jobsriver's five-second forceCancelGrace, where the recording phase alone already left zero margin. It costs a forced drain up to two extra seconds of waiting on a sweep worker, and only once a format wires an observer that actually blocks. healthObserveTimeout's doc points at the arithmetic for whoever changes either value.
• No metric, log or span is added at this seam. An operator cannot yet tell a monitor with no observer from one whose probes never ran. The plan scopes emitters to the format packages, so closing that belongs with step 4 rather than here.

Diff size

790 reviewable LOC, past the 500 the development model asks about. Splitting would not help, because the split is almost entirely test code:

Group	LOC
Tests: health_test.go 573, healthjob_test.go 56, connection_test_handler_integration_test.go 2	631
Production: internal/remote/health.go 146, cmd/artifact-registry/wire_remote.go 13	159

Governing ADRs

Checked, and one applies as a conformance rather than a deviation.

• ADR-023 is the relevant one, and this seam is the shape it prescribes: the interface is declared in the shared internal/remote package, implemented per format, and injected by the composition root, exactly as the FormatCollector shape at docs/adr/023_code_structure_and_enforcement.md:289 does. internal/remote gains no format import.
• ADR-012 rules itself out at :89, which states that LabKit's operational Prometheus metrics are "not part of the usage data collection decision". This step emits nothing anyway.
• No deviation, so no handbook amendment MR is needed. The local docs/adr/ mirror was read rather than trusted by title. ./scripts/adr-freshness.sh exits 1: the mirror is stale by two upstream commits, both on 007_database_schema.md (a maven_packages tombstone index and the size_bytes decrement), neither of which bears on a diff with no schema change, no SQL and no migration. The one internal ADR is not readable from this environment, so treat its conformance as unverified rather than clear.

Testing

Unit tests only; no migration, no SQL, no schema change, so nothing here needs a database.

Covered: all four settled outcome shapes, both RequestSent arms, the no-host marker on a builder refusal, silence on abandonment and on the three pre-round-trip refusals, both triggers, a nil observer changing no durable write and no audit event, the panicking observer contained with the write and the audit event surviving, the report context arriving detached, a request carrying no URL, all three non-recording shapes HealthProbeResult.Recorded names (counter degraded, failed write, and lost compare-and-set), the report context carrying a deadline of its own, and the observer driven through the real sweep composition, which is the only place the concurrent ObserveProbe path runs.

Run locally on the head commit:

• go build ./..., go vet ./..., gofmt -l clean
• go test -race -count=2 ./internal/remote and go test ./cmd/artifact-registry/... pass
• scripts/ci/check-comment-caps.sh --base origin/main passes
• Guardrail 7: internal/managementapi/connection_test_handler_integration_test.go opens with //go:build integration, which CI lint never compiles, so golangci-lint run --build-tags=integration --max-same-issues=0 --max-issues-per-linter=0 --uniq-by-line=false ./internal/remote/... ./internal/managementapi/... ./cmd/artifact-registry/... was run. It reports 11 issues, all of them pre-existing and none in a file this MR touches: wire_npm_boot_integration_test.go, wire_npm_observability_integration_test.go, wire_root_dispatcher_boot_integration_test.go, and healthjob_integration_test.go.

E2e scenario impact: none. e2e.npm.remote.packument-cold-miss, -cache-hit and -revalidate in docs/testing/e2e/npm.md all exist and none is affected: no format implements the observer, so no served response, status, or cache verdict changes.

Spec coverage

Spec: docs/specs/S15-npm-remote.md

Scope of this MR: the internal/remote probe-observer seam. No format implements the hook yet, so every npm-side emitter row below is still open.

Acceptance criteria

#	Criterion	Tests
P-1	Packument cache miss	Owned by the merged packument-route MRs. Untouched here.
P-2	dist.tarball rewrite	Owned by the merged packument-route MRs. Untouched here.
P-3	dist.shasum/dist.integrity preserved	Owned by the merged packument-route MRs. Untouched here.
P-4	Abbreviated packument	Owned by the merged packument-route MRs. Untouched here.
P-5	Fresh hit	Owned by the merged packument-route MRs. Untouched here.
P-6	Stale hit, etag matches	Owned by the merged packument-route MRs. Untouched here.
P-7	Stale hit, etag differs	Owned by the merged packument-route MRs. Untouched here.
P-8	New upstream version surfaces after the window	Owned by the merged packument-route MRs. Untouched here.
P-9	Dist-tags document	Owned by the merged dist-tags-route MRs. Untouched here.
P-10	Tag install resolves client-side	Owned by the merged dist-tags-route MRs. Untouched here.
P-11	Single-flight on a metadata read	Owned by the merged packument-route MRs. Untouched here.
P-12	Single-flight leader failure	Owned by the merged packument-route MRs. Untouched here.
T-1	Tarball cache miss, fill	Owned by the merged tarball-route MRs. Untouched here.
T-2	Fresh tarball cache hit	Owned by the merged tarball-route MRs. Untouched here.
T-3	HEAD on a cache miss fills	Owned by the merged tarball-route MRs. Untouched here.
T-4	Pinned upstream never revalidates	Owned by the merged tarball-route MRs. Untouched here.
T-5	Stale tarball hit, etag matches	Owned by the merged tarball-route MRs. Untouched here.
T-6	Stale tarball hit, upstream re-published	Owned by the merged tarball-route MRs. Untouched here.
T-7	Stale tarball hit, no stored ETag	Owned by the merged tarball-route MRs. Untouched here.
T-8	Tarball relayed unverified	Owned by the merged tarball-route MRs. Untouched here.
T-9	Single-flight on a tarball read	Owned by the merged tarball-route MRs. Untouched here.
T-10	Path-traversal filename rejected	Owned by the merged tarball-route MRs. Untouched here.
T-11	Unparseable filename rejected	Owned by the merged tarball-route MRs. Untouched here.
T-12	Outbound segment safety	Owned by the merged tarball-route MRs. Untouched here.
T-13	A cache serve carries a content validator	Owned by the merged tarball-route MRs. Untouched here.
T-14	Every serve shape is cache-separated	Owned by the merged tarball-route MRs. Untouched here.
T-15	A client conditional request is answered	Owned by the merged tarball-route MRs. Untouched here.
W-1	Publish rejected	Owned by the merged write-unavailability MR. Untouched here.
W-2	Dist-tag mutation rejected	Owned by the merged write-unavailability MR. Untouched here.
W-3	Unpublish routes rejected	Owned by the merged write-unavailability MR. Untouched here.
C-1	Bearer attached	Owned by npm's HealthTargetResolver and request-builder suites. Untouched here.
C-2	Credential redaction	Owned by the npm credential-store suites. Untouched here.
C-3	Credential clearing on URL change	Owned by the S13 credential-manager suites. Untouched here.
C-4	On-demand probe: HEAD within probe_timeout	Probe mechanics stay with TestHealthMonitor_Probe_RequestDiscipline. This MR adds that the on-demand trigger reports too: TestHealthMonitor_Probe_ObservesBothTriggers/manual.
E-1	Upstream 404 propagates	Owned by the merged read-route MRs. Untouched here.
E-2	Transport failure with cache	Owned by the merged read-route MRs. Untouched here.
E-3	Transport failure without cache	Owned by the merged read-route MRs. Untouched here.
E-4	Upstream 5xx not a fallback trigger	Owned by the merged read-route MRs. Untouched here.

Error cases

#	Condition	Tests
EC-1	Write verb on a remote repository: 405	Owned by the merged write-unavailability MR. Untouched here.
EC-2	Upstream 404 for a packument: 404 package_not_found	Owned by the merged packument-route MRs. Untouched here.
EC-3	Upstream 404 for a tarball: 404 file_not_found	Owned by the merged tarball-route MRs. Untouched here.
EC-4	Metadata document over max_remote_packument_size	Owned by the merged packument-route MRs. Untouched here.
EC-5	Transport failure, cache entry exists	Owned by the merged read-route MRs. Untouched here.
EC-6	Transport failure, no cache entry: 503	Owned by the merged read-route MRs. Untouched here.
EC-7	Coalesced follower: wait timeout or leader lost: 503	Owned by the merged read-route MRs. Untouched here.
EC-8	Upstream 5xx propagated	Owned by the merged read-route MRs. Untouched here. On the probe path a 5xx is a failed probe, observed by TestHealthMonitor_Probe_ObservesEveryProbeOutcome/5xx_is_a_failed_probe.
EC-9	Upstream non-404 4xx: upstream_rejected	Owned by the merged read-route MRs. Untouched here.
EC-10	Upstream status outside 4xx and 5xx: 503	Owned by the merged read-route MRs. Untouched here.
EC-11	URL-decoded {file_name} with .. or /: 400	Owned by the merged tarball-route MRs. Untouched here.
EC-12	{file_name} does not parse: 404	Owned by the merged tarball-route MRs. Untouched here.
EC-13	Package name fails the npm regex: 422	Owned by the merged read-route MRs. Untouched here.
EC-14	Validators on a proxied metadata read	Owned by the merged metadata-route MRs. Untouched here.

Security considerations

#	Concern	Tests
SC-1	Credential hygiene: no token in logs or metrics	The observation carries no credential-bearing value: TestHealthMonitor_Probe_ObservesEveryProbeOutcome asserts the whole struct, whose only upstream-derived field is the host. The wide-event side is step 4's.
SC-2	Credentials at rest	Owned by the S13 credential store. Untouched here.
SC-3	SSRF	Owned by the S13 upstream client. This MR adds no new outbound call.
SC-4	Cross-origin redirect token stripping	Owned by the S13 upstream client. Untouched here.
SC-5	Outbound path-segment safety	Owned by the read routes and by TestHealthMonitor_Probe_RejectsNonRelativeHealthPath, which this MR extends with TestHealthMonitor_Probe_RefusalsBeforeTheRoundTripObserveNothing/health_path_is_not_relative: a refused path reports nothing either.
SC-6	Tarball-URL rewrite keeps clients on the AR	Owned by the merged packument-route MRs. Untouched here.
SC-7	Tarball integrity is client-enforced	Owned by the merged tarball-route MRs. Untouched here.
SC-8	Error-payload hygiene	Owned by the merged read-route MRs. Untouched here. The observation is not client-visible.

Observability surface

Spec row	Tests
Health probe: reachable taxonomy, a 401/403/404 is healthy	TestHealthMonitor_Probe_ObservesEveryProbeOutcome/404_counts_as_reachable pins the reported Passed; the durable half stays with TestHealthMonitor_Probe_ReachableStatuses.
Emitter coverage: npm_remote_upstream_responses_total{npm_operation="probe"}	Not closed here. This MR lands the seam the emitter needs; the npm emitter is the next step of this plan.
Emitter coverage: npm_remote_proxy operation probe	Not closed here, same reason.
Emitter coverage: npm_remote_cache_events_total, npm_remote_cache_fill_bytes, field coalesced	Out of this MR's scope; the metadata-route step of this plan owns them.
Emitter coverage: npm_remote_proxy field version	Out of this MR's scope; the field-retirement step of this plan owns it.

Seam behavior this MR pins beyond the spec text

The plan fixes where the report sits and what the silent arms are; the spec does not speak to either. Recorded here so a reviewer can check the reading.

Behavior	Tests
One observation per settled probe, carrying identity, host, status, Passed	TestHealthMonitor_Probe_ObservesEveryProbeOutcome
A BuildRequest failure that is not abandonment reports the no-host marker	TestHealthMonitor_Probe_ObservesBuildFailureWithNoHostMarker
An abandoned probe reports nothing	TestHealthMonitor_Probe_AbandonmentObservesNothing
A refusal before the round trip reports nothing	TestHealthMonitor_Probe_RefusalsBeforeTheRoundTripObserveNothing
The report follows the round trip, not the durable write	TestHealthMonitor_Probe_ObservesWhatNothingDurableRecords
A nil observer changes no durable write and no audit event	TestHealthMonitor_Probe_ObserverLeavesTheProbeUnchanged
Scheduled and manual triggers both report	TestHealthMonitor_Probe_ObservesBothTriggers
The report context carries a deadline of its own	TestHealthMonitor_Probe_BoundsTheReport

Resolved spec ambiguities: none. The spec fixes the probe taxonomy and the plan fixes the hook's shape and placement, so no assertion here rested on a reading that needed choosing.

Related to #800 (closed)