chore(jobs): create job series at startup (#1595 stack: 4/4)

What this delivers

This MR is part 4 of 4 of the split of the original !3348 (merged) change. The original diff is now in four stacked MRs, and this MR keeps the last part: the job series. A new instance now exports the job series of every registered kind at 0 before the kind first runs.

Phase 1 of the issue covers HTTP routes, named database queries and job kinds. Phase 2 (S06, S20, S22, the format packages, authentication and authorization, and the Redis caches) moves to a separate issue.

Fix. jobs.Metrics.InitKind creates the job children for one kind from a jobs.KindSeries, with WithLabelValues and no observation. jobsasynq.Client.InitMetrics and jobsriver.Client.InitMetrics call it for every registered kind. wireJobs calls each one after the last registration of its backend.

Family asynq kind River kind River periodic kind
jobs_duration_seconds, jobs_in_flight Yes Yes Yes
jobs_processed_total at ok, error, discard Yes Yes Yes
jobs_processed_total at cancelled No Only for a CancelOnPanic kind Only for a CancelOnPanic kind
jobs_enqueued_total Yes Yes No
jobs_errors_total at payload_too_large, backend_unavailable, unknown Yes Yes No

Specific behavior:

  • asynq never books cancelled. River books cancelled only for a CancelOnPanic kind.
  • A River periodic kind gets no jobs_enqueued_total or jobs_errors_total children, because the River scheduler inserts it without Enqueue.
  • River kinds get their series only after migrations are ready, which is when River can work them.
  • No job series exist when no jobs backend is wired.
  • jobs_errors_total{error="unknown_kind"} and the asynq unregistered kind still appear only on first use.

docs/dev/observability.md now lists the job families that get series at startup.

The stack

Part MR Scope
1 !3353 (merged) HTTP server series
2 !3354 (merged) Datastore query series
3 !3355 (merged) Storage query series
4 This MR Job series

Depends on !3355 (merged), and through it on !3353 (merged) and !3354 (merged). Merge the parts in order. The tree at the head of this MR is the same as the head of this MR before the split, 9f5b5285608815c320547be1552405c410687eb9. git diff 9f5b5285608815c320547be1552405c410687eb9 HEAD is empty.

Spec coverage

Source: the issue card acceptance list. No spec governs startup series creation; S03-B and S27 govern only the label values.

# Criterion Tests
AC-1 The owning package exposes an Init helper; the call sits after the registration TestMetrics_InitKind_CreatesTheKindChildrenWithoutAnObservation, TestClient_InitMetrics_SeedsEveryRegisteredKind (jobsriver), TestClient_InitMetrics_SeedsEveryRegisteredHandler (jobsasynq), TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup
AC-2 Histogram children are created without an observation TestMetrics_InitKind_CreatesTheKindChildrenWithoutAnObservation, both TestClient_InitMetrics_*
AC-5 Job kinds come from the registered kinds, per wired backend both TestClient_InitMetrics_*, TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup
AC-6 Conditional families are seeded only under their emitter condition TestWiring_SeedsHTTPQueryAndJobSeriesWithNoTraffic (no job series with no backend wired), TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup (River polled after migration readiness)
AC-8 AuditCardinality passes with the seeded series present TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup (full seeds)
AC-9 Phase 1 dashboard panels show 0 on staging with no traffic Not repository-testable: manual verification after merge.
AC-11 Startup cost measured and recorded in the MR Not repository-testable: see "Startup cost" below.

The two backend InitMetrics tests assert the full jobs_errors_total set, so they fail if an unknown_kind child is seeded. The issue card lists no error case and no security consideration.

E2E scenario catalog

No scenario is added or affected. Part 1 (!3353 (merged)) changes the explanation of the row e2e.management.observability.metric-families-registered.

Size

The diff is 547 added and 10 removed lines in 12 files, measured against part 3.

File group Files Added Removed
Go tests (*_test.go) 7 453 9
Production Go 4 89 0
docs/dev/ 1 5 1

The production change is 89 lines. The tests are most of the diff, and they cover one family, so a further split does not make the review smaller.

Startup cost

Acceptance item 11 was measured on 2026-10-15 on one local host, at 9f5b5285608815c320547be1552405c410687eb9, with the run-artifact-registry driver and no client traffic. That tree is the result of all four parts.

Series count:

Boot origin/main All four parts Delta
Warm 466 18,121 +17,655
Cold, sample 1 484 18,139 +17,655
Cold, sample 2 484 18,139 +17,655

Warm scrape with all four parts, by family prefix:

Family prefix Series
gitlab_artifact_registry_http_ 6,464
gitlab_artifact_registry_database_quer 10,908
gitlab_artifact_registry_jobs_ 326

At origin/main, these three prefixes have 43 series in total.

Time to ready, in seconds from process start:

Boot origin/main All four parts
Warm 0.96 0.97
Cold, sample 1 29.39 66.67
Cold, sample 2 44.96 79.22

The warm boots differ by 0.01 s, which is less than the 0.1 s resolution of the process start. The migrations take most of the cold boot time. Two samples for each sha do not separate a branch effect from host noise, so this measurement does not attribute the cold difference to the change.

What remains

  • Item 9: the Phase 1 dashboard panels on staging with no traffic are not verified yet.
  • Phase 2 moves to a separate issue.

No customer-visible surface changes, so docs/user/ is not affected.

🤖 Generated with Claude Code

Closes #1595 (closed)

Edited by Pawel Rozlach

Merge request reports

Loading
Loading