Loading
feat(usagedata): add the S23 tracker, contexts, and drop metric (Step 2)
What
Step 2 of the S23 usage data foundation plan: the internal/usagedata package.
Tracker(New/Track/Start/Shutdown, anapp.Component): no-op when disabled, fire-and-forgetTrackthat never blocks, and a flush-and-waitShutdownbounded by ctx (workaround for labkit#130:Emitter.Stopcancels the final drain POST).- Both custom contexts on every event:
gitlab_standard/1-1-8(static config template + per-event correlation and organization) andartifact_registry_context/1-0-0(mirrors the iglu schema field-for-field;ar_instance_versionalways fromapp.Version). - Pre-enqueue validation: invalid events drop with
gitlab_artifact_registry_usagedata_dropped_events_total{event,reason}+ one Warn log naming the event, offending field, and correlation ID; LabKit's emitter metrics register undergitlab_artifact_registry_snowplow_events_*. - Delivery-failure onset is logged once per unhealthy period, so a misconfigured collector (for example a URL missing its
/com.snowplowanalytics.snowplow/tp2path) surfaces in logs, not only in metrics.
Stacked on !910 (merged) (Step 1 config fragment): this MR targets the Step 1 branch, so the diff here is Step 2 only. It retargets to main automatically when !910 (merged) merges.
Spec coverage
Spec: docs/specs/S23-usage-data-collection.md
Acceptance criteria
| # | Criterion | Tests |
|---|---|---|
| AC-1 | Disabled or omitted section: Track/lifecycle succeed, no HTTP | TestNew_Disabled_Noop |
| AC-2 | Collector POST cx carries both context schema URIs with fields |
TestTracker_Track_EmitsBothContexts, TestTracker_Track_OmitsUnsetOptionalFields |
| AC-3 | ar_instance_version = app.Version, not settable via Attrs |
TestTracker_Track_InstanceVersionFromBuildInfo, TestAttrs_FieldSurface |
| AC-4 | correlation_id equals the request's correlation ID |
TestTracker_Track_EmitsBothContexts |
| AC-5 | Out-of-enum RepositoryKind/UpstreamType drops + counter |
TestTracker_Track_InvalidEnum_DropsEvent |
| AC-6 | Forced gitlab_standard failure drops through the same path |
TestTracker_Track_GitLabStandardFailure_DropsEvent |
| AC-7 | Shutdown drains queued events; bounded when collector unreachable | TestTracker_Shutdown_DrainsQueuedEvents, TestTracker_Shutdown_UnreachableCollector_ReturnsWithinBudget |
| AC-8 | Track safe under concurrent use; no events lost | TestTracker_Track_ConcurrentEmission (runs under -race in test:race) |
| AC-9 | S17 create emits exactly one first event | Step 5 scope. Not tested in this MR. |
| AC-10 | Emitter metrics + dropped counter on the service registry | Step 3 scope. Not tested in this MR. Metric names pinned here via droppedMetric. |
Error cases
| # | Condition | Tests |
|---|---|---|
| E-1 | Invalid config: load fails, service refuses to start | Step 1 (this branch): internal/config/usagedata_test.go |
| E-2 | Collector unreachable: 1s-backoff retry, 10k ring buffer, Track never blocks |
LabKit-emitter-owned. Shutdown budget under an unreachable collector: TestTracker_Shutdown_UnreachableCollector_ReturnsWithinBudget |
| E-3 | Context validation failure: drop + counter + log | TestTracker_Track_InvalidEnum_DropsEvent, TestTracker_Track_GitLabStandardFailure_DropsEvent |
| E-4 | Track after Shutdown: no-op, counter reason="stopped" |
TestTracker_TrackAfterShutdown_DropsWithStoppedReason |
| E-5 | Process exit with queued events: lost, acceptable | Accepted by design (ADR-012); no unit-level assertion possible. |
Security considerations
| # | Concern | Tests |
|---|---|---|
| S-1 | No tokens/credentials/PII; identity limited to organization_id | TestTracker_Track_NoIdentityOrRailsScopedFields |
| S-2 | Props carries only catalog dimensions |
Runtime key allow-list: TestTracker_Track_UnknownPropKey_DropsEvent, TestValidateProps; round-trip pinned in TestTracker_Track_EmitsBothContexts. |
| S-3 | Attrs cannot carry version, correlation, or identity |
TestAttrs_FieldSurface, TestTracker_Track_InstanceVersionFromBuildInfo |
| S-4 | TLS enforced for the collector outside development | Step 1 config CEL: internal/config/usagedata_test.go |
| S-5 | enabled = false guarantees nothing leaves the process |
TestNew_Disabled_Noop |
Resolved spec ambiguities
- Design
New(cfg Config, ...)->config.UsageDataConfig(the Step 1 runtime struct built for this tracker;brandlistvalidator.Newprecedent). - Configuration "deployment_type is derived from realm" -> identity
mapping except
saas->.com(the only pairing LabKit'sDeploymentTypeenum admits). - Design "empty fields are omitted from the emitted context" -> applies
to every zero-valued optional
Attrsfield (enums, strings,uuid.Nil,OrganizationID0), not only enums;CacheHitnil omits,falseemits. - Observability drop log (event name, offending field, correlation ID)
-> emitted through the
log.FromContext(ctx)seam the request path provides (auth middleware / managementapi precedent).
Notes
internal/usagedata/context_internal_test.gois extra white-box coverage not named in the plan's Step 2 Tests entry (helpers + enum-membership acceptance).- The live iglu schema types
formatas a plain string; the spec's enum is enforced byValidate(the comments state this). - The plan's Dependencies section still says LabKit v2.28.0;
go.modpins v2.29.2 (renovate bump after the plan merged; theevents/snowplowpackage is unchanged). The flush-and-wait workaround stays until labkit#130 ships. - New metric catalog rows for
docs/dev/observability.mdland with Step 3 (wiring), when the metrics first reach/metrics. - The first commit is the documented test-first
--no-verifycarve-out; every other commit ran the full hook chain.
Related to #272 (closed)
Edited by João Pereira