chore(metrics): seed metric series at startup (#1628 stack: 5/5)

Summary

Problem. A Prometheus vector publishes nothing until its first child exists. A pod that has served no traffic therefore omits the family from /-/metrics, and the dashboard panels for storage, lifecycle, accounting, the format packages, auth and the Redis-backed caches read "No data". An operator cannot tell that state apart from a broken emitter.

Impact. Operators and on-call see empty panels for these areas until real traffic books each label set. On a quiet deployment that can take hours, or never. No package client sees a change.

Change. Each package in scope now creates its series at startup through an Init… helper beside its RegisterMetrics. The composition root calls each helper after registration. The rule is: a family whose emitter a tier builds conditionally is seeded in that tier, under the same condition, or not at all. initServiceSeries seeds the families that every production pod wires. In it, the authz denial reasons follow the authorization posture, and the auth JWKS fetch pair is seeded only under the token-exchange validator. Each conditional gate is a small function beside its call site, with a table test on both arms:

  • storageSeriesFor: the storage driver, the URL cache and the CDN children of the assembled stack, with the URL-cache request children only when the storage.url_cache block does not set dry_run: true and no CDN block without an IP filter wraps the cache, and the CDN ip_filtered bypass child only when ip_filtered_by in the CDN block selects an IP filter,
  • authzEnforced: the authz decision-layer denial reasons under the enforcing posture,
  • iamCacheEventsSeeded: the IAM relationship caches where iam.cache_ttl is positive and a cache store is wired,
  • authSeriesFor: the JWKS fetch pair under the token-exchange validator,
  • glazSeriesSeeded: the GLAZ client under the enforcing posture,
  • mavenRemoteSeriesSeeded: the Maven remote series where an upstream client exists,
  • ociRemoteSeriesFor: the OCI remote series for the wired blob store and upstream client.

The purger series have no gate function. wireLifecycle seeds them inside the closure that registers the periodic purge jobs, so only a pod whose River client runs those jobs carries them.

A helper seeds only the label values that production can book. The token_handler surface gets no missing_credential failure child, because the auth middleware answers a credential-less token request under v2_auth_token. The storage seed creates retry_type="custom" only for the GCS driver, because the S3 driver books only native.

Two seeding rules come from review findings M1 and M2:

  • M1, authz_denials_total by posture. authz.InitMetrics takes the enforced flag alone. Every pod gets unresolved_repository and both organization_ancestor_skipped_total skip reasons. Only an enforcing pod gets the six decision-layer reasons. No pod gets anonymous, because an enforcing boot always wires the token-exchange validator, which never returns an anonymous identity. No pod gets glaz_denied, because no built-in role books it.
  • M2, the CDN bypass child under an IP filter. StorageSeries gains CDNIPFiltered. storageSeriesFor sets it from the CDN block's ip_filtered_by, normalized as the cloudcdn middleware reads it: any value but "" and none selects a filter. Only then does the seed create storage_cdn_redirects_total{bypass="true",bypass_reason="ip_filtered"}, because the default none builds no filter and nothing can book that child.

The helper creates the children with no observation, so a counter or gauge reads 0 and a histogram exports _count 0. docs/dev/observability.md lists the families that startup now carries and the ones that still appear on first use.

Downsides. The seeding is a fixed set of WithLabelValues calls at boot and does no I/O, so the added startup cost is bounded by the number of seeded series. The measurement is acceptance item 8 below. The seeded label sets raise the service's metric cardinality, and AuditCardinality holds them to the pinned sets.

Checklist

  • Ran it: The changed packages' tests, the pre-commit chain, and gate-run --changed --integration-tests: build, vet, lint, gofmt and test pass, and lint-integration reports only pre-existing findings in files this diff does not touch. The one integration-tagged file the diff edits did not complete locally: both -tags=integration attempts hung inside testcontainers' container teardown, with no assertion failure, so CI runs it. Beyond the tests, the service was booted and /-/metrics scraped with no traffic, and the seeded families were present at zero. No package client was driven, because the change adds no route, RPC or client-visible behavior.
  • Interfaces and data: None. No API, configuration, schema, wire or storage format changes, and no stored data is affected. The new exported symbols are the per-package Init… helpers and their option structs, all inside internal/ and cmd/. docs/dev/observability.md is updated in this MR. docs/user/** is not, because no customer-visible surface changes. No spec governs the change.
  • Failure: The seeding runs once at boot, returns no error and does no I/O. A label list that disagrees with the vector's declaration panics at the WithLabelValues call, and the tests cover every seeded set. No request path changes, so no new error, timeout or partial write appears.
  • Rollout: Nothing to migrate, and a previous release against this configuration behaves as before. During a rolling deploy, a pod that carries the change seeds the families and an older pod does not, so a seeded family can appear once a new pod is scraped. docs/dev/observability.md records that reading.
  • Hot path: None. The seeding is a fixed set of WithLabelValues calls at boot, before the server starts, and it adds no query, allocation or unbounded work to a request path.
  • Database: n/a — the change adds no migration and changes no query.
  • Security: n/a — the change adds no input, route, permission or audited action, and it does not change how callers are authenticated or authorized. Every seeded label value is a static string from the code.
  • ADRs: None governs this change. I checked docs/adr/ for a constraint on metric registration, startup seeding or package placement. The change adds no route, format, storage or data-path behavior, and each family stays in the package that already owns it.
  • E2E: None added or changed. The change adds no route, RPC or protocol behavior, so no scenario's journey changes. e2e.management.observability.metric-families-registered (docs/testing/e2e/management.md) stays true as written: it asserts the registered sentinels on /-/metrics, and this change seeds children of families that are already registered.
  • Simplicity: The shape is not a choice here. Acceptance item 1 of the issue and docs/dev/observability.md fix it as an Init… helper beside each package's RegisterMetrics, called from the composition root. The simplification taken is the other way: one initServiceSeries call covers the families that every production pod wires, and the authz denial reasons and the auth JWKS pair carry their own conditions there. Every other conditioned family keeps a call site at its tier.

Acceptance coverage

The rows map the acceptance items of #1628 (closed) to the tests on this branch. No spec governs: the issue's acceptance items are the source of truth, and the Init... convention is in docs/dev/observability.md.

Acceptance item (#1628 (closed)) Test
1. Each Phase 2 package exposes an Init... helper beside its RegisterMetrics, called from the composition root after registration TestInitStorageMetrics_CreatesTheWiredStackChildren (storage), TestInitMetrics_CreatesThePurgerSeriesWithoutAnObservation (lifecycle), TestInitMetrics_CreatesTheCoreSeriesWithoutAnObservation (accounting), TestInitMetrics_CreatesTheNegativeCacheChildrenWithoutAnObservation (remote), TestInitRelationshipCacheEvents_CreatesTheEventChildrenWithoutAnObservation (iam), TestInitMetrics_CreatesTheDispatchChildrenWithoutAnObservation (oci, npm, maven), TestInitRemoteMetrics_SeedsOnlyTheWiredEmitters and TestInitVirtualMetrics_SeedsTheRoutedOperations (oci), TestInitRemoteAndVirtualMetrics_SeedTheReachableChildren (npm, maven), TestInitMetrics_CreatesTheAuthenticationChildrenWithoutAnObservation (auth), TestInitMetrics_CreatesTheDenialChildrenWithoutAnObservation (authz), TestInitMetrics_CreatesTheGLAZSeriesWithoutAnObservation (glaz); the call sites by TestWiring_SeedsServiceSeriesWithNoTraffic and TestMountSlugAnchoredFormats_SeedsNpmResolutionsPerPosture
2. Phase 2 histogram children are created without an observation; a test asserts _count == 0 straight after initialisation the drift bytes and rows assertions in TestWiring_SeedsServiceSeriesWithNoTraffic, and the negative-cache duration assertion in TestInitMetrics_CreatesTheNegativeCacheChildrenWithoutAnObservation
3. Phase 2 conditional families are seeded only under their emitter's condition the composition-root gates, each on both arms: TestStorageSeriesFor_FollowsTheWiredBlocks (including the CDN ip_filtered_by rows), TestAuthzEnforced_FollowsThePosture, TestIAMCacheEventsSeeded_OnlyWithAPositiveCacheTTLAndAStore, TestAuthSeriesFor_SeedsTheJWKSPairOnlyUnderTokenExchange, TestGLAZSeriesSeeded_OnlyUnderTheEnforcingPosture, TestMavenRemoteSeriesSeeded_OnlyWithAnUpstreamClient, TestOCIRemoteSeriesFor_FollowsTheStoreAndDoerGates; the per-package conditional cases: storage driver (retry_type="custom" for GCS only), URL-cache and CDN cases (the ip_filtered bypass child with and without an IP filter) in TestInitStorageMetrics_CreatesTheWiredStackChildren, the two posture rows of TestInitMetrics_CreatesTheDenialChildrenWithoutAnObservation (authz), the unenforced fixture of TestWiring_SeedsServiceSeriesWithNoTraffic, which expects only unresolved_repository, auth's TokenExchange case, iam's unwired cache case, TestInitRemoteMetrics_SeedsOnlyTheWiredEmitters (oci) and TestInitResolutionMetrics_SeedsReplayedOnlyWhenTheHolderIsInstalled (npm)
4. Phase 2 series are visible on /-/metrics of a freshly started instance with no traffic TestWiring_MetricsEndpointServesSeededSeriesWithNoTraffic (DB-less exposition) and the exposition block of TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup (live DB)
5. AuditCardinality passes with the Phase 2 seeded series present the metrics.AuditCardinality assertions in TestWiring_SeedsServiceSeriesWithNoTraffic and in TestInitMetrics_CreatesThePurgerSeriesWithoutAnObservation
6. Dashboard panels show 0 rather than "No data" on staging with no traffic (manual, after a release) no test: needs a release that carries the change
7. docs/dev/observability.md lists the Phase 2 families created at startup and the ones still on first use no test: a documentation change, on this branch in docs/dev/observability.md
8. The added startup cost of the Phase 2 seeding is measured and recorded in the MR no test: a manual measurement

Size

At cc106735c, the diff against origin/main is 46 files, 2,938 added and 31 deleted lines: 832 added lines of Go production code (154 in cmd/artifact-registry, 678 in the internal/ packages), 2,059 added lines of Go tests, and 47 added lines of docs/dev/.

Merge order

Five open merge requests conflict with this branch, and each is clean against main. Whichever merge request lands second resolves the conflict. Nothing was adopted or resolved here.

  • !3458 (merged) conflicts on internal/accounting/metrics_test.go and on docs/dev/observability.md. In the test file, both branches add a test block at one anchor, so the resolution keeps both blocks. !3458 (merged) is the live one: open, non-Draft, clean against main and in review.
  • !3452 (merged) conflicts on docs/dev/observability.md, where this branch rewrites the lifecycle rows.
  • !3380 (merged) conflicts on docs/dev/observability.md.
  • !3460 (merged) conflicts on internal/accounting/metrics_test.go and internal/remote/metrics_test.go, and !2971 (merged) on internal/auth/metrics_test.go. Each is a content conflict at one anchor in a test file, where the sibling adds its own test block and this branch adds the Init… seeding tests. !3460 (merged) and !2971 (merged) are Draft and not live, and !2971 (merged) is stacked on an unmerged branch.

!2969 (merged) retires the authorization posture from the composition root. If !2969 (merged) merges first, authzEnforced and the posture wording in docs/dev/observability.md (the authz_denials_total row and the startup list) need an update to match.

Acceptance items that remain open

  • Item 6 — the post-release dashboard check. The Phase 2 panels must show 0 rather than "No data" on staging with no traffic. This needs a release that carries the change, so it is checked after that release, on a freshly started pod.
  • Item 8 — the added startup cost of the seeding. The seeding is a fixed set of WithLabelValues calls at boot and does no I/O, so it adds no request-path or database work. This item is a manual measurement.

Remaining: the two manual acceptance items above, the post-release dashboard check and the startup-cost measurement.

Stack

This merge request is part 5 of 5 of one reviewed change (vespian_gl/issue-1628-metric-initialization-phase-2-storage), split into a stack so each part is reviewable on its own. Merge the parts in this order, bottom first:

  1. !3478 (merged) chore(metrics): add seeders for storage and jobs (#1628 (closed) stack: 1/5)
  2. !3479 (merged) chore(metrics): add seeders for auth and authz (#1628 (closed) stack: 2/5)
  3. !3480 (merged) chore(metrics): add seeders for maven and npm (#1628 (closed) stack: 3/5)
  4. !3481 (merged) chore(metrics): add seeders for oci families (#1628 (closed) stack: 4/5)
  5. !3470 (merged) chore(metrics): seed metric series at startup (#1628 (closed) stack: 5/5) (this merge request)

It targets !3481 (merged), the part below it, which merges first. It is the top of the stack and merges last.

Related to #1628 (closed)

Edited by Pawel Rozlach

Merge request reports

Loading
Loading