feat(maven): observability metrics + wide event (S14 plan: 17/19)

Summary

Step 17 of the Maven remote plan: the three gitlab_artifact_registry_maven_remote_* collectors and the maven_remote_proxy wide event. Artifact, metadata, and sidecar read paths emit at completion. Operators can see cache class, upstream outcome (including cache_error), fill size, and client-facing bytes. Credentials, coordinates, and upstream hosts stay off metric labels.

Collectors register through the existing mavenCollectors / RegisterMetrics seam. One request is one cache event (miss / hit_fresh / hit_stale / revalidated), matching the wide event's cache field. An error response after a stale lookup is miss. coalesced is true when FetchResult.Coalesced is set (SingleFlight follower). Fill bytes and upstream counters count the leader only. Probe callers stay uninstrumented; probe is reserved on the event only. The wide event is assembled in-package and is not yet an S03-A typed emitter.

Spec amendments in this MR

These are the Observability-section edits this step owns. They keep the spec aligned with the collectors and the catalog:

  • cache_error is a closed upstream outcome and now also covers ErrSingleFlightTimeout / ErrLeaderCanceled.
  • cache_fill_bytes and upstream_responses_total count the SingleFlight leader only.
  • Cache label is cache_path (not path) so S15/S16 can keep their own vocabularies.
  • revalidated is part of the cache vocabulary; error-after-stale is miss.
  • Wide-event field is status (fields.HTTPStatusCode), not http_status.
  • outcome, duration_s, and coalesced are on the event; bytes is 0 on 304, 302, and HEAD.
  • Failure arms that never fetch still emit maven_remote_proxy with cache=miss.

Plan-file Status table edits stay on this branch from earlier docs(plans) commits. A later docs(plans) batch-fill can own adjacent rows if Step 18 is in flight.

Governing ADRs

  • ADR-006 — LabKit-prefixed Prometheus registerer and structured logs; collectors ride the existing maven.RegisterMetrics call.
  • ADR-023 — emitters stay in internal/format/maven; no new package, no database/sql.
  • Checked and not engaged: ADR-001, 002, 005, 007, 008, 009, 010, 012, 022.

Per Mirror freshness and internal ADRs, the local mirror is not the authority; no deviation is introduced here.

Spec coverage

Spec: docs/specs/S14-maven-remote.md

Specs live under docs/specs/; acceptance criteria, error cases, and security considerations are the source of truth for what the tests assert (docs/dev/go-testing.md).

This step owns the Observability section. Artifact, metadata, sidecar, write, credential, and error-mapping ACs landed in Steps 11–16.

Observability

# Criterion Tests
O-1 Three prefixed collectors, spec types and label sets TestMavenRemoteMetrics_RegisteredExposesAllThree, TestMavenRemoteMetrics_Types, TestMavenRemoteMetrics_LabelSets, TestRegisterMetrics_ExposedViaGatherer
O-2 Every label value from a closed enum TestMavenRemoteMetrics_OperationEnumClosed, TestMavenRemoteMetrics_OutcomeEnumClosed, TestMavenRemoteMetrics_PathEnumClosed, TestMavenRemoteMetrics_EventEnumClosed, TestMavenRemoteMetrics_FillPathEnumClosed, TestMavenRemoteMetrics_ProxyOperationEnumClosed, TestMavenRemoteMetrics_HelpersEnforceClosedEnums, TestMavenRemoteMetrics_EmitEnforcesClosedEnums
O-3 No credential, namespace_id, coordinate, or upstream host on a metric label TestMavenRemoteMetrics_NoHighCardinalityOrCredentialLabels, TestMavenRemoteMetrics_CardinalityAuditPasses
O-4 cache_error is a closed upstream outcome TestMavenRemoteMetrics_OutcomeEnumClosed, TestMavenRemoteUpstreamOutcome_Classifies
O-5 Fill histogram uses body-size buckets; counts committed leader fill bytes only TestMavenRemoteMetrics_FillBytesHistogramBuckets
O-6 One cache event per request, same vocabulary as the wide-event cache field TestMavenRemoteCacheEventFunnels, TestMavenRemoteWideCacheEventMapping
O-7 maven_remote_proxy carries S03 baseline plus artifact / metadata / sidecar fields TestMavenRemoteProxyWideEvent_BaselineFields, TestMavenRemoteProxyWideEvent_ArtifactFields, TestMavenRemoteProxyWideEvent_MetadataFields, TestMavenRemoteProxyWideEvent_SidecarFields, TestMavenRemoteProxyWideEvent_CompletionEmitsAtTerminalStatus
O-8 Wide-event outcome / status (fields.HTTPStatusCode) / maven_code / bytes mapping TestMavenRemoteCodeForTerminal_Maps, TestMavenRemoteProxyWideEvent_CompletionEmitsAtTerminalStatus
O-9 coalesced is a bool from FetchResult.Coalesced TestMavenRemoteProxyWideEvent_CoalescedFieldIsBool
O-10 Unparseable upstream URL renders the audit invalid-url marker TestMavenRemoteUpstreamHost_Markers
O-11 Catalog rows in docs/dev/observability.md reviewed as docs; no Go test

probe on the wide event is reserved for S13/S17 probe callers. Those callers are out of this plan's Scope.

Error cases

# Condition Tests
Observability does not own HTTP error mapping Steps 13–16. This MR classifies the same outcomes for counters (TestMavenRemoteUpstreamOutcome_Classifies) and the wide event (TestMavenRemoteCodeForTerminal_Maps, TestMavenRemoteProxyWideEvent_CompletionEmitsAtTerminalStatus).

Security considerations

Concern Tests
Credential hygiene TestMavenRemoteMetrics_NoHighCardinalityOrCredentialLabels, TestMavenRemoteProxyWideEvent_NoCredential
Credentials at rest, SSRF, redirect stripping, outbound path safety S13/S17-owned. Not tested in this MR.
Error-payload hygiene Wide event may carry upstream_host; client responses do not. Client hygiene is S13-owned and unchanged.

docs/testing/ (Guardrail 12)

No e2e scenario is added or affected. The Maven e2e catalog covers hosted local repositories only. This step pins registration, enums, and the wide-event field set in internal/format/maven/remote_metrics_test.go.

Diff footprint (Guardrail 18)

+2007 −61 total against main. The plan carries this as one step (Step 17). Splitting would separate collectors from the emit seams that make them true. By file group:

  • Implementation: remote_metrics.go (+610), remote_artifact.go / remote_metadata.go / remote_sidecar.go / remote_serve.go / metrics.go / cardinality.go — emit seams and registration
  • Tests: remote_metrics_test.go (+1205), buffered_test.go — registerer and field-set pins
  • Docs: docs/dev/observability.md, docs/specs/S14-maven-remote.md

Tests dominate the added lines. The production surface is one new file plus emit calls at the existing read seams.

Related to #286 (closed)

Edited by Moaz Khalifa

Merge request reports

Loading
Loading