Publish startup migration duration, applied count and advisory-lock wait, with a completion log record
Split from #1606 (closed) (gap 5). Parent epic: &24088. Blocks #1664 (ArtifactRegistryDatabaseMigrationNearBudget). Absorbs #1380 (closed) (closed as absorbed by this issue).
Decisions (operator ruling 2026-10-25)
- Q6: this issue absorbs #1380 (closed). One MR adds the migration metrics and #1380 (closed)'s structured completion log record in
Runner.Start, because both editinternal/datastore/migrations/runner.go:172and its surroundings. - Q7: this MR amends S03-B itself. #1718 (closed) lands first.
Problem
The startup migration runs goose behind pg_try_advisory_lock, which it retries 300 times at 1 s each, all under a 5-minute upTimeout. It publishes no duration, no applied count and no lock wait. A migration that grows toward the 5-minute ceiling is invisible until a rollout crash-loops. A replica that waits for the lock looks the same as one that is migrating.
From #1380 (closed): Runner.Start (internal/datastore/migrations/runner.go:128) logs only on failure. The partial-failure path emits migration partially applied (runner.go:196), and the success path returns nil silently. goose's own per-migration completion record is gated behind goose.WithVerbose, which migrations.go never enables. A successful run produces no structured record of how long it took or which versions applied.
Evidence
Research base: origin/main at e2c420456, 2026-10-25.
internal/datastore/migrations/migrations.go:40-43:lock.NewPostgresSessionLocker(lock.WithLockTimeout(1, 300), …).internal/datastore/migrations/runner.go:28:upTimeout = 5 * time.Minute. At:172,_, err = r.up(upCtx, r.client.DB(), r.logger)discards the[]*goose.MigrationResult, which carries aDurationper migration.internal/datastore/postgres.go:45-56: the migration client is a separatepostgres.Clientwith a pool of one connection. It has no pool collector.cmd/artifact-registry/wire_database.go:79-94builds the runner and passes it no registerer.cmd/artifact-registry/migration_runner.go:58-76: the pass runs in a goroutine while the server serves/-/metrics, so a gauge set on success is scrapeable. A failed pass callsonFataland the process exits, so a failure-only series would rarely be scraped. The crash loop is already covered byKubeContainersWaitingInError(#1664 notes).
Fix direction
- Pass a
prometheus.Registerer(the prefixedappReg) intomigrations.NewRunnerfromwire_database.go. - Publish:
database_migration_last_duration_seconds: a gauge with the wall time of the last successfulUp, lock wait included. #1664's sketch already readsgitlab_artifact_registry_database_migration_last_duration_seconds.database_migration_last_lock_wait_seconds: a gauge with the time spent inSessionLock, measured by a thin wrapper around goose'slock.SessionLocker.database_migrations_applied_total: a counter of the migrations this process applied, so it is 0 on a no-op boot.- Optional:
database_migration_last_success_timestamp_seconds. Seed every series at registration, so a no-op boot publishes 0 and not an absent series.
- In the same place, emit the #1380 (closed) structured completion record on every successful
Up: duration, applied count and versions, and final state. Log around ther.upcall from the returned[]*goose.MigrationResult, or enablegoose.WithVerbose. - Update the S03-B
### Databasetable,docs/dev/observability.md, and S04'sAdvisory lockingsection if it describes observability. S04 owns the advisory-lock contract thatrunner.go:24-27cites.
Acceptance criteria
- After a successful startup pass, the three series show the pass's duration, its lock wait and the number of migrations applied. A no-op boot shows 0 applied and a small duration.
- An integration test holds the advisory lock from another session, then releases it, and asserts that the lock-wait gauge is at least the held time.
- Every successful
Upemits one structured completion log record with duration, applied versions and final state (#1380 (closed)'s ask). A test asserts it. - The cardinality audit passes. The S03-B table and
docs/dev/observability.mdlist the families. - #1664's
MigrationNearBudgetsketch is checked against the shipped names (a comment on #1664).
Size and fit
About 450-600 Go lines including the integration test and the log record. This fits one smurfit unit.
Dependencies
- None blocking. Unblocks #1664 (
MigrationNearBudget). - Related: #1410 (the table-size ceiling against
upTimeout). - Related: #1569 (the Migrator switch) removes the boot-time runner (
runner.go,migration_runner.go). If #1569 is scheduled before this issue starts, re-scope this issue to the Migrator Job before implementing it; the metrics and the log record would not survive the switch as written.