feat(usagedata): add the S23 tracker, contexts, and drop metric (Step 2)

What

Step 2 of the S23 usage data foundation plan: the internal/usagedata package.

  • Tracker (New/Track/Start/Shutdown, an app.Component): no-op when disabled, fire-and-forget Track that never blocks, and a flush-and-wait Shutdown bounded by ctx (workaround for labkit#130: Emitter.Stop cancels the final drain POST).
  • Both custom contexts on every event: gitlab_standard/1-1-8 (static config template + per-event correlation and organization) and artifact_registry_context/1-0-0 (mirrors the iglu schema field-for-field; ar_instance_version always from app.Version).
  • Pre-enqueue validation: invalid events drop with gitlab_artifact_registry_usagedata_dropped_events_total{event,reason} + one Warn log naming the event, offending field, and correlation ID; LabKit's emitter metrics register under gitlab_artifact_registry_snowplow_events_*.
  • Delivery-failure onset is logged once per unhealthy period, so a misconfigured collector (for example a URL missing its /com.snowplowanalytics.snowplow/tp2 path) surfaces in logs, not only in metrics.

Stacked on !910 (merged) (Step 1 config fragment): this MR targets the Step 1 branch, so the diff here is Step 2 only. It retargets to main automatically when !910 (merged) merges.

Spec coverage

Spec: docs/specs/S23-usage-data-collection.md

Acceptance criteria

# Criterion Tests
AC-1 Disabled or omitted section: Track/lifecycle succeed, no HTTP TestNew_Disabled_Noop
AC-2 Collector POST cx carries both context schema URIs with fields TestTracker_Track_EmitsBothContexts, TestTracker_Track_OmitsUnsetOptionalFields
AC-3 ar_instance_version = app.Version, not settable via Attrs TestTracker_Track_InstanceVersionFromBuildInfo, TestAttrs_FieldSurface
AC-4 correlation_id equals the request's correlation ID TestTracker_Track_EmitsBothContexts
AC-5 Out-of-enum RepositoryKind/UpstreamType drops + counter TestTracker_Track_InvalidEnum_DropsEvent
AC-6 Forced gitlab_standard failure drops through the same path TestTracker_Track_GitLabStandardFailure_DropsEvent
AC-7 Shutdown drains queued events; bounded when collector unreachable TestTracker_Shutdown_DrainsQueuedEvents, TestTracker_Shutdown_UnreachableCollector_ReturnsWithinBudget
AC-8 Track safe under concurrent use; no events lost TestTracker_Track_ConcurrentEmission (runs under -race in test:race)
AC-9 S17 create emits exactly one first event Step 5 scope. Not tested in this MR.
AC-10 Emitter metrics + dropped counter on the service registry Step 3 scope. Not tested in this MR. Metric names pinned here via droppedMetric.

Error cases

# Condition Tests
E-1 Invalid config: load fails, service refuses to start Step 1 (this branch): internal/config/usagedata_test.go
E-2 Collector unreachable: 1s-backoff retry, 10k ring buffer, Track never blocks LabKit-emitter-owned. Shutdown budget under an unreachable collector: TestTracker_Shutdown_UnreachableCollector_ReturnsWithinBudget
E-3 Context validation failure: drop + counter + log TestTracker_Track_InvalidEnum_DropsEvent, TestTracker_Track_GitLabStandardFailure_DropsEvent
E-4 Track after Shutdown: no-op, counter reason="stopped" TestTracker_TrackAfterShutdown_DropsWithStoppedReason
E-5 Process exit with queued events: lost, acceptable Accepted by design (ADR-012); no unit-level assertion possible.

Security considerations

# Concern Tests
S-1 No tokens/credentials/PII; identity limited to organization_id TestTracker_Track_NoIdentityOrRailsScopedFields
S-2 Props carries only catalog dimensions Runtime key allow-list: TestTracker_Track_UnknownPropKey_DropsEvent, TestValidateProps; round-trip pinned in TestTracker_Track_EmitsBothContexts.
S-3 Attrs cannot carry version, correlation, or identity TestAttrs_FieldSurface, TestTracker_Track_InstanceVersionFromBuildInfo
S-4 TLS enforced for the collector outside development Step 1 config CEL: internal/config/usagedata_test.go
S-5 enabled = false guarantees nothing leaves the process TestNew_Disabled_Noop

Resolved spec ambiguities

  • Design New(cfg Config, ...) -> config.UsageDataConfig (the Step 1 runtime struct built for this tracker; brandlistvalidator.New precedent).
  • Configuration "deployment_type is derived from realm" -> identity mapping except saas -> .com (the only pairing LabKit's DeploymentType enum admits).
  • Design "empty fields are omitted from the emitted context" -> applies to every zero-valued optional Attrs field (enums, strings, uuid.Nil, OrganizationID 0), not only enums; CacheHit nil omits, false emits.
  • Observability drop log (event name, offending field, correlation ID) -> emitted through the log.FromContext(ctx) seam the request path provides (auth middleware / managementapi precedent).

Notes

  • internal/usagedata/context_internal_test.go is extra white-box coverage not named in the plan's Step 2 Tests entry (helpers + enum-membership acceptance).
  • The live iglu schema types format as a plain string; the spec's enum is enforced by Validate (the comments state this).
  • The plan's Dependencies section still says LabKit v2.28.0; go.mod pins v2.29.2 (renovate bump after the plan merged; the events/snowplow package is unchanged). The flush-and-wait workaround stays until labkit#130 ships.
  • New metric catalog rows for docs/dev/observability.md land with Step 3 (wiring), when the metrics first reach /metrics.
  • The first commit is the documented test-first --no-verify carve-out; every other commit ran the full hook chain.

Related to #272 (closed)

Edited by João Pereira

Merge request reports

Loading
Loading