chore(jobs): create job series at startup (#1595 stack: 4/4)
What this delivers
This MR is part 4 of 4 of the split of the original !3348 (merged) change.
The original diff is now in four stacked MRs, and this MR keeps the last part: the job series.
A new instance now exports the job series of every registered kind at 0 before the kind first runs.
Phase 1 of the issue covers HTTP routes, named database queries and job kinds. Phase 2 (S06, S20, S22, the format packages, authentication and authorization, and the Redis caches) moves to a separate issue.
Fix.
jobs.Metrics.InitKind creates the job children for one kind from a jobs.KindSeries, with WithLabelValues and no observation.
jobsasynq.Client.InitMetrics and jobsriver.Client.InitMetrics call it for every registered kind.
wireJobs calls each one after the last registration of its backend.
| Family | asynq kind | River kind | River periodic kind |
|---|---|---|---|
jobs_duration_seconds, jobs_in_flight |
Yes | Yes | Yes |
jobs_processed_total at ok, error, discard |
Yes | Yes | Yes |
jobs_processed_total at cancelled |
No | Only for a CancelOnPanic kind |
Only for a CancelOnPanic kind |
jobs_enqueued_total |
Yes | Yes | No |
jobs_errors_total at payload_too_large, backend_unavailable, unknown |
Yes | Yes | No |
Specific behavior:
- asynq never books
cancelled. River bookscancelledonly for aCancelOnPanickind. - A River periodic kind gets no
jobs_enqueued_totalorjobs_errors_totalchildren, because the River scheduler inserts it withoutEnqueue. - River kinds get their series only after migrations are ready, which is when River can work them.
- No job series exist when no jobs backend is wired.
jobs_errors_total{error="unknown_kind"}and the asynqunregisteredkind still appear only on first use.
docs/dev/observability.md now lists the job families that get series at startup.
The stack
| Part | MR | Scope |
|---|---|---|
| 1 | !3353 (merged) | HTTP server series |
| 2 | !3354 (merged) | Datastore query series |
| 3 | !3355 (merged) | Storage query series |
| 4 | This MR | Job series |
Depends on !3355 (merged), and through it on !3353 (merged) and !3354 (merged).
Merge the parts in order.
The tree at the head of this MR is the same as the head of this MR before the split, 9f5b5285608815c320547be1552405c410687eb9.
git diff 9f5b5285608815c320547be1552405c410687eb9 HEAD is empty.
Spec coverage
Source: the issue card acceptance list. No spec governs startup series creation; S03-B and S27 govern only the label values.
| # | Criterion | Tests |
|---|---|---|
| AC-1 | The owning package exposes an Init helper; the call sits after the registration | TestMetrics_InitKind_CreatesTheKindChildrenWithoutAnObservation, TestClient_InitMetrics_SeedsEveryRegisteredKind (jobsriver), TestClient_InitMetrics_SeedsEveryRegisteredHandler (jobsasynq), TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup |
| AC-2 | Histogram children are created without an observation | TestMetrics_InitKind_CreatesTheKindChildrenWithoutAnObservation, both TestClient_InitMetrics_* |
| AC-5 | Job kinds come from the registered kinds, per wired backend | both TestClient_InitMetrics_*, TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup |
| AC-6 | Conditional families are seeded only under their emitter condition | TestWiring_SeedsHTTPQueryAndJobSeriesWithNoTraffic (no job series with no backend wired), TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup (River polled after migration readiness) |
| AC-8 | AuditCardinality passes with the seeded series present |
TestIntegration_SeedsMountedRoutesAndJobKindsAtStartup (full seeds) |
| AC-9 | Phase 1 dashboard panels show 0 on staging with no traffic | Not repository-testable: manual verification after merge. |
| AC-11 | Startup cost measured and recorded in the MR | Not repository-testable: see "Startup cost" below. |
The two backend InitMetrics tests assert the full jobs_errors_total set, so they fail if an unknown_kind child is seeded.
The issue card lists no error case and no security consideration.
E2E scenario catalog
No scenario is added or affected.
Part 1 (!3353 (merged)) changes the explanation of the row e2e.management.observability.metric-families-registered.
Size
The diff is 547 added and 10 removed lines in 12 files, measured against part 3.
| File group | Files | Added | Removed |
|---|---|---|---|
Go tests (*_test.go) |
7 | 453 | 9 |
| Production Go | 4 | 89 | 0 |
docs/dev/ |
1 | 5 | 1 |
The production change is 89 lines. The tests are most of the diff, and they cover one family, so a further split does not make the review smaller.
Startup cost
Acceptance item 11 was measured on 2026-10-15 on one local host, at 9f5b5285608815c320547be1552405c410687eb9, with the run-artifact-registry driver and no client traffic.
That tree is the result of all four parts.
Series count:
| Boot | origin/main | All four parts | Delta |
|---|---|---|---|
| Warm | 466 | 18,121 | +17,655 |
| Cold, sample 1 | 484 | 18,139 | +17,655 |
| Cold, sample 2 | 484 | 18,139 | +17,655 |
Warm scrape with all four parts, by family prefix:
| Family prefix | Series |
|---|---|
gitlab_artifact_registry_http_ |
6,464 |
gitlab_artifact_registry_database_quer |
10,908 |
gitlab_artifact_registry_jobs_ |
326 |
At origin/main, these three prefixes have 43 series in total.
Time to ready, in seconds from process start:
| Boot | origin/main | All four parts |
|---|---|---|
| Warm | 0.96 | 0.97 |
| Cold, sample 1 | 29.39 | 66.67 |
| Cold, sample 2 | 44.96 | 79.22 |
The warm boots differ by 0.01 s, which is less than the 0.1 s resolution of the process start. The migrations take most of the cold boot time. Two samples for each sha do not separate a branch effect from host noise, so this measurement does not attribute the cold difference to the change.
What remains
- Item 9: the Phase 1 dashboard panels on staging with no traffic are not verified yet.
- Phase 2 moves to a separate issue.
No customer-visible surface changes, so docs/user/ is not affected.
Closes #1595 (closed)