chore(metrics): seed metric series at startup (#1628 stack: 5/5)
Summary
Problem. A Prometheus vector publishes nothing until its first child exists. A pod that has served no traffic therefore omits the family from /-/metrics, and the dashboard panels for storage, lifecycle, accounting, the format packages, auth and the Redis-backed caches read "No data". An operator cannot tell that state apart from a broken emitter.
Impact. Operators and on-call see empty panels for these areas until real traffic books each label set. On a quiet deployment that can take hours, or never. No package client sees a change.
Change. Each package in scope now creates its series at startup through an Init… helper beside its RegisterMetrics. The composition root calls each helper after registration. The rule is: a family whose emitter a tier builds conditionally is seeded in that tier, under the same condition, or not at all. initServiceSeries seeds the families that every production pod wires. In it, the authz denial reasons follow the authorization posture, and the auth JWKS fetch pair is seeded only under the token-exchange validator. Each conditional gate is a small function beside its call site, with a table test on both arms:
storageSeriesFor: the storage driver, the URL cache and the CDN children of the assembled stack, with the URL-cache request children only when thestorage.url_cacheblock does not setdry_run: trueand no CDN block without an IP filter wraps the cache, and the CDNip_filteredbypass child only whenip_filtered_byin the CDN block selects an IP filter,authzEnforced: the authz decision-layer denial reasons under the enforcing posture,iamCacheEventsSeeded: the IAM relationship caches whereiam.cache_ttlis positive and a cache store is wired,authSeriesFor: the JWKS fetch pair under the token-exchange validator,glazSeriesSeeded: the GLAZ client under the enforcing posture,mavenRemoteSeriesSeeded: the Maven remote series where an upstream client exists,ociRemoteSeriesFor: the OCI remote series for the wired blob store and upstream client.
The purger series have no gate function. wireLifecycle seeds them inside the closure that registers the periodic purge jobs, so only a pod whose River client runs those jobs carries them.
A helper seeds only the label values that production can book. The token_handler surface gets no missing_credential failure child, because the auth middleware answers a credential-less token request under v2_auth_token. The storage seed creates retry_type="custom" only for the GCS driver, because the S3 driver books only native.
Two seeding rules come from review findings M1 and M2:
- M1,
authz_denials_totalby posture.authz.InitMetricstakes the enforced flag alone. Every pod getsunresolved_repositoryand bothorganization_ancestor_skipped_totalskip reasons. Only an enforcing pod gets the six decision-layer reasons. No pod getsanonymous, because an enforcing boot always wires the token-exchange validator, which never returns an anonymous identity. No pod getsglaz_denied, because no built-in role books it. - M2, the CDN bypass child under an IP filter.
StorageSeriesgainsCDNIPFiltered.storageSeriesForsets it from the CDN block'sip_filtered_by, normalized as thecloudcdnmiddleware reads it: any value but""andnoneselects a filter. Only then does the seed createstorage_cdn_redirects_total{bypass="true",bypass_reason="ip_filtered"}, because the defaultnonebuilds no filter and nothing can book that child.
The helper creates the children with no observation, so a counter or gauge reads 0 and a histogram exports _count 0. docs/dev/observability.md lists the families that startup now carries and the ones that still appear on first use.
Downsides. The seeding is a fixed set of WithLabelValues calls at boot and does no I/O, so the added startup cost is bounded by the number of seeded series. The measurement is acceptance item 8 below. The seeded label sets raise the service's metric cardinality, and AuditCardinality holds them to the pinned sets.
Checklist
- Ran it: The changed packages' tests, the pre-commit chain, and
gate-run --changed --integration-tests: build, vet, lint, gofmt and test pass, andlint-integrationreports only pre-existing findings in files this diff does not touch. The one integration-tagged file the diff edits did not complete locally: both-tags=integrationattempts hung inside testcontainers' container teardown, with no assertion failure, so CI runs it. Beyond the tests, the service was booted and/-/metricsscraped with no traffic, and the seeded families were present at zero. No package client was driven, because the change adds no route, RPC or client-visible behavior. - Interfaces and data: None. No API, configuration, schema, wire or storage format changes, and no stored data is affected. The new exported symbols are the per-package
Init…helpers and their option structs, all insideinternal/andcmd/.docs/dev/observability.mdis updated in this MR.docs/user/**is not, because no customer-visible surface changes. No spec governs the change. - Failure: The seeding runs once at boot, returns no error and does no I/O. A label list that disagrees with the vector's declaration panics at the
WithLabelValuescall, and the tests cover every seeded set. No request path changes, so no new error, timeout or partial write appears. - Rollout: Nothing to migrate, and a previous release against this configuration behaves as before. During a rolling deploy, a pod that carries the change seeds the families and an older pod does not, so a seeded family can appear once a new pod is scraped.
docs/dev/observability.mdrecords that reading. - Hot path: None. The seeding is a fixed set of
WithLabelValuescalls at boot, before the server starts, and it adds no query, allocation or unbounded work to a request path. - Database: n/a — the change adds no migration and changes no query.
- Security: n/a — the change adds no input, route, permission or audited action, and it does not change how callers are authenticated or authorized. Every seeded label value is a static string from the code.
- ADRs: None governs this change. I checked
docs/adr/for a constraint on metric registration, startup seeding or package placement. The change adds no route, format, storage or data-path behavior, and each family stays in the package that already owns it. - E2E: None added or changed. The change adds no route, RPC or protocol behavior, so no scenario's journey changes.
e2e.management.observability.metric-families-registered(docs/testing/e2e/management.md) stays true as written: it asserts the registered sentinels on/-/metrics, and this change seeds children of families that are already registered. - Simplicity: The shape is not a choice here. Acceptance item 1 of the issue and
docs/dev/observability.mdfix it as anInit…helper beside each package'sRegisterMetrics, called from the composition root. The simplification taken is the other way: oneinitServiceSeriescall covers the families that every production pod wires, and the authz denial reasons and the auth JWKS pair carry their own conditions there. Every other conditioned family keeps a call site at its tier.
Acceptance coverage
The rows map the acceptance items of #1628 (closed) to the tests on this branch. No spec governs: the issue's acceptance items are the source of truth, and the Init... convention is in docs/dev/observability.md.
| Acceptance item (#1628 (closed)) | Test |
|---|---|
1. Each Phase 2 package exposes an Init... helper beside its RegisterMetrics, called from the composition root after registration |
TestInitStorageMetrics_CreatesTheWiredStackChildren (storage), TestInitMetrics_CreatesThePurgerSeriesWithoutAnObservation (lifecycle), TestInitMetrics_CreatesTheCoreSeriesWithoutAnObservation (accounting), TestInitMetrics_CreatesTheNegativeCacheChildrenWithoutAnObservation (remote), TestInitRelationshipCacheEvents_CreatesTheEventChildrenWithoutAnObservation (iam), TestInitMetrics_CreatesTheDispatchChildrenWithoutAnObservation (oci, npm, maven), TestInitRemoteMetrics_SeedsOnlyTheWiredEmitters and TestInitVirtualMetrics_SeedsTheRoutedOperations (oci), TestInitRemoteAndVirtualMetrics_SeedTheReachableChildren (npm, maven), TestInitMetrics_CreatesTheAuthenticationChildrenWithoutAnObservation (auth), TestInitMetrics_CreatesTheDenialChildrenWithoutAnObservation (authz), TestInitMetrics_CreatesTheGLAZSeriesWithoutAnObservation (glaz); the call sites by TestWiring_SeedsServiceSeriesWithNoTraffic and TestMountSlugAnchoredFormats_SeedsNpmResolutionsPerPosture |
2. Phase 2 histogram children are created without an observation; a test asserts _count == 0 straight after initialisation |
the drift bytes and rows assertions in TestWiring_SeedsServiceSeriesWithNoTraffic, and the negative-cache duration assertion in TestInitMetrics_CreatesTheNegativeCacheChildrenWithoutAnObservation |
| 3. Phase 2 conditional families are seeded only under their emitter's condition | the composition-root gates, each on both arms: TestStorageSeriesFor_FollowsTheWiredBlocks (including the CDN ip_filtered_by rows), TestAuthzEnforced_FollowsThePosture, TestIAMCacheEventsSeeded_OnlyWithAPositiveCacheTTLAndAStore, TestAuthSeriesFor_SeedsTheJWKSPairOnlyUnderTokenExchange, TestGLAZSeriesSeeded_OnlyUnderTheEnforcingPosture, TestMavenRemoteSeriesSeeded_OnlyWithAnUpstreamClient, TestOCIRemoteSeriesFor_FollowsTheStoreAndDoerGates; the per-package conditional cases: storage driver (retry_type="custom" for GCS only), URL-cache and CDN cases (the ip_filtered bypass child with and without an IP filter) in TestInitStorageMetrics_CreatesTheWiredStackChildren, the two posture rows of TestInitMetrics_CreatesTheDenialChildrenWithoutAnObservation (authz), the unenforced fixture of TestWiring_SeedsServiceSeriesWithNoTraffic, which expects only unresolved_repository, auth's TokenExchange case, iam's unwired cache case, TestInitRemoteMetrics_SeedsOnlyTheWiredEmitters (oci) and TestInitResolutionMetrics_SeedsReplayedOnlyWhenTheHolderIsInstalled (npm) |
4. Phase 2 series are visible on /-/metrics of a freshly started instance with no traffic |
TestWiring_MetricsEndpointServesSeededSeriesWithNoTraffic (DB-less exposition) and the exposition block of TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup (live DB) |
5. AuditCardinality passes with the Phase 2 seeded series present |
the metrics.AuditCardinality assertions in TestWiring_SeedsServiceSeriesWithNoTraffic and in TestInitMetrics_CreatesThePurgerSeriesWithoutAnObservation |
| 6. Dashboard panels show 0 rather than "No data" on staging with no traffic (manual, after a release) | no test: needs a release that carries the change |
7. docs/dev/observability.md lists the Phase 2 families created at startup and the ones still on first use |
no test: a documentation change, on this branch in docs/dev/observability.md |
| 8. The added startup cost of the Phase 2 seeding is measured and recorded in the MR | no test: a manual measurement |
Size
At cc106735c, the diff against origin/main is 46 files, 2,938 added and 31 deleted lines: 832 added lines of Go production code (154 in cmd/artifact-registry, 678 in the internal/ packages), 2,059 added lines of Go tests, and 47 added lines of docs/dev/.
Merge order
Five open merge requests conflict with this branch, and each is clean against main. Whichever merge request lands second resolves the conflict. Nothing was adopted or resolved here.
- !3458 (merged) conflicts on
internal/accounting/metrics_test.goand ondocs/dev/observability.md. In the test file, both branches add a test block at one anchor, so the resolution keeps both blocks. !3458 (merged) is the live one: open, non-Draft, clean againstmainand in review. - !3452 (merged) conflicts on
docs/dev/observability.md, where this branch rewrites the lifecycle rows. - !3380 (merged) conflicts on
docs/dev/observability.md. - !3460 (merged) conflicts on
internal/accounting/metrics_test.goandinternal/remote/metrics_test.go, and !2971 (merged) oninternal/auth/metrics_test.go. Each is a content conflict at one anchor in a test file, where the sibling adds its own test block and this branch adds theInit…seeding tests. !3460 (merged) and !2971 (merged) are Draft and not live, and !2971 (merged) is stacked on an unmerged branch.
!2969 (merged) retires the authorization posture from the composition root. If !2969 (merged) merges first, authzEnforced and the posture wording in docs/dev/observability.md (the authz_denials_total row and the startup list) need an update to match.
Acceptance items that remain open
- Item 6 — the post-release dashboard check. The Phase 2 panels must show
0rather than "No data" on staging with no traffic. This needs a release that carries the change, so it is checked after that release, on a freshly started pod. - Item 8 — the added startup cost of the seeding. The seeding is a fixed set of
WithLabelValuescalls at boot and does no I/O, so it adds no request-path or database work. This item is a manual measurement.
Remaining: the two manual acceptance items above, the post-release dashboard check and the startup-cost measurement.
Stack
This merge request is part 5 of 5 of one reviewed change (vespian_gl/issue-1628-metric-initialization-phase-2-storage), split into a stack so each part is reviewable on its own. Merge the parts in this order, bottom first:
- !3478 (merged) chore(metrics): add seeders for storage and jobs (#1628 (closed) stack: 1/5)
- !3479 (merged) chore(metrics): add seeders for auth and authz (#1628 (closed) stack: 2/5)
- !3480 (merged) chore(metrics): add seeders for maven and npm (#1628 (closed) stack: 3/5)
- !3481 (merged) chore(metrics): add seeders for oci families (#1628 (closed) stack: 4/5)
- !3470 (merged) chore(metrics): seed metric series at startup (#1628 (closed) stack: 5/5) (this merge request)
It targets !3481 (merged), the part below it, which merges first. It is the top of the stack and merges last.
Related to #1628 (closed)