feat(maven): observability metrics + wide event (S14 plan: 17/19)
Summary
Step 17 of the Maven remote plan: the three gitlab_artifact_registry_maven_remote_* collectors and the maven_remote_proxy wide event. Artifact, metadata, and sidecar read paths emit at completion. Operators can see cache class, upstream outcome (including cache_error), fill size, and client-facing bytes. Credentials, coordinates, and upstream hosts stay off metric labels.
Collectors register through the existing mavenCollectors / RegisterMetrics seam. One request is one cache event (miss / hit_fresh / hit_stale / revalidated), matching the wide event's cache field. An error response after a stale lookup is miss. coalesced is true when FetchResult.Coalesced is set (SingleFlight follower). Fill bytes and upstream counters count the leader only. Probe callers stay uninstrumented; probe is reserved on the event only. The wide event is assembled in-package and is not yet an S03-A typed emitter.
Spec amendments in this MR
These are the Observability-section edits this step owns. They keep the spec aligned with the collectors and the catalog:
cache_erroris a closed upstream outcome and now also coversErrSingleFlightTimeout/ErrLeaderCanceled.cache_fill_bytesandupstream_responses_totalcount the SingleFlight leader only.- Cache label is
cache_path(notpath) so S15/S16 can keep their own vocabularies. revalidatedis part of thecachevocabulary; error-after-stale ismiss.- Wide-event field is
status(fields.HTTPStatusCode), nothttp_status. outcome,duration_s, andcoalescedare on the event;bytesis 0 on 304, 302, and HEAD.- Failure arms that never fetch still emit
maven_remote_proxywithcache=miss.
Plan-file Status table edits stay on this branch from earlier docs(plans) commits. A later docs(plans) batch-fill can own adjacent rows if Step 18 is in flight.
Governing ADRs
- ADR-006 — LabKit-prefixed Prometheus registerer and structured logs; collectors ride the existing
maven.RegisterMetricscall. - ADR-023 — emitters stay in
internal/format/maven; no new package, nodatabase/sql. - Checked and not engaged: ADR-001, 002, 005, 007, 008, 009, 010, 012, 022.
Per Mirror freshness and internal ADRs, the local mirror is not the authority; no deviation is introduced here.
Spec coverage
Spec: docs/specs/S14-maven-remote.md
Specs live under docs/specs/; acceptance criteria, error cases, and security considerations are the source of truth for what the tests assert (docs/dev/go-testing.md).
This step owns the Observability section. Artifact, metadata, sidecar, write, credential, and error-mapping ACs landed in Steps 11–16.
Observability
| # | Criterion | Tests |
|---|---|---|
| O-1 | Three prefixed collectors, spec types and label sets | TestMavenRemoteMetrics_RegisteredExposesAllThree, TestMavenRemoteMetrics_Types, TestMavenRemoteMetrics_LabelSets, TestRegisterMetrics_ExposedViaGatherer |
| O-2 | Every label value from a closed enum | TestMavenRemoteMetrics_OperationEnumClosed, TestMavenRemoteMetrics_OutcomeEnumClosed, TestMavenRemoteMetrics_PathEnumClosed, TestMavenRemoteMetrics_EventEnumClosed, TestMavenRemoteMetrics_FillPathEnumClosed, TestMavenRemoteMetrics_ProxyOperationEnumClosed, TestMavenRemoteMetrics_HelpersEnforceClosedEnums, TestMavenRemoteMetrics_EmitEnforcesClosedEnums |
| O-3 | No credential, namespace_id, coordinate, or upstream host on a metric label |
TestMavenRemoteMetrics_NoHighCardinalityOrCredentialLabels, TestMavenRemoteMetrics_CardinalityAuditPasses |
| O-4 | cache_error is a closed upstream outcome |
TestMavenRemoteMetrics_OutcomeEnumClosed, TestMavenRemoteUpstreamOutcome_Classifies |
| O-5 | Fill histogram uses body-size buckets; counts committed leader fill bytes only | TestMavenRemoteMetrics_FillBytesHistogramBuckets |
| O-6 | One cache event per request, same vocabulary as the wide-event cache field |
TestMavenRemoteCacheEventFunnels, TestMavenRemoteWideCacheEventMapping |
| O-7 | maven_remote_proxy carries S03 baseline plus artifact / metadata / sidecar fields |
TestMavenRemoteProxyWideEvent_BaselineFields, TestMavenRemoteProxyWideEvent_ArtifactFields, TestMavenRemoteProxyWideEvent_MetadataFields, TestMavenRemoteProxyWideEvent_SidecarFields, TestMavenRemoteProxyWideEvent_CompletionEmitsAtTerminalStatus |
| O-8 | Wide-event outcome / status (fields.HTTPStatusCode) / maven_code / bytes mapping |
TestMavenRemoteCodeForTerminal_Maps, TestMavenRemoteProxyWideEvent_CompletionEmitsAtTerminalStatus |
| O-9 | coalesced is a bool from FetchResult.Coalesced |
TestMavenRemoteProxyWideEvent_CoalescedFieldIsBool |
| O-10 | Unparseable upstream URL renders the audit invalid-url marker |
TestMavenRemoteUpstreamHost_Markers |
| O-11 | Catalog rows in docs/dev/observability.md |
reviewed as docs; no Go test |
probe on the wide event is reserved for S13/S17 probe callers. Those callers are out of this plan's Scope.
Error cases
| # | Condition | Tests |
|---|---|---|
| — | Observability does not own HTTP error mapping | Steps 13–16. This MR classifies the same outcomes for counters (TestMavenRemoteUpstreamOutcome_Classifies) and the wide event (TestMavenRemoteCodeForTerminal_Maps, TestMavenRemoteProxyWideEvent_CompletionEmitsAtTerminalStatus). |
Security considerations
| Concern | Tests |
|---|---|
| Credential hygiene | TestMavenRemoteMetrics_NoHighCardinalityOrCredentialLabels, TestMavenRemoteProxyWideEvent_NoCredential |
| Credentials at rest, SSRF, redirect stripping, outbound path safety | S13/S17-owned. Not tested in this MR. |
| Error-payload hygiene | Wide event may carry upstream_host; client responses do not. Client hygiene is S13-owned and unchanged. |
docs/testing/ (Guardrail 12)
No e2e scenario is added or affected. The Maven e2e catalog covers hosted local repositories only. This step pins registration, enums, and the wide-event field set in internal/format/maven/remote_metrics_test.go.
Diff footprint (Guardrail 18)
+2007 −61 total against main. The plan carries this as one step (Step 17). Splitting would separate collectors from the emit seams that make them true. By file group:
- Implementation:
remote_metrics.go(+610),remote_artifact.go/remote_metadata.go/remote_sidecar.go/remote_serve.go/metrics.go/cardinality.go— emit seams and registration - Tests:
remote_metrics_test.go(+1205),buffered_test.go— registerer and field-set pins - Docs:
docs/dev/observability.md,docs/specs/S14-maven-remote.md
Tests dominate the added lines. The production surface is one new file plus emit calls at the existing read seams.
Related to #286 (closed)