Publish startup migration duration, applied count and advisory-lock wait, with a completion log record

Split from #1606 (closed) (gap 5). Parent epic: &24088. Blocks #1664 (ArtifactRegistryDatabaseMigrationNearBudget). Absorbs #1380 (closed) (closed as absorbed by this issue).

Decisions (operator ruling 2026-10-25)

  • Q6: this issue absorbs #1380 (closed). One MR adds the migration metrics and #1380 (closed)'s structured completion log record in Runner.Start, because both edit internal/datastore/migrations/runner.go:172 and its surroundings.
  • Q7: this MR amends S03-B itself. #1718 (closed) lands first.

Problem

The startup migration runs goose behind pg_try_advisory_lock, which it retries 300 times at 1 s each, all under a 5-minute upTimeout. It publishes no duration, no applied count and no lock wait. A migration that grows toward the 5-minute ceiling is invisible until a rollout crash-loops. A replica that waits for the lock looks the same as one that is migrating.

From #1380 (closed): Runner.Start (internal/datastore/migrations/runner.go:128) logs only on failure. The partial-failure path emits migration partially applied (runner.go:196), and the success path returns nil silently. goose's own per-migration completion record is gated behind goose.WithVerbose, which migrations.go never enables. A successful run produces no structured record of how long it took or which versions applied.

Evidence

Research base: origin/main at e2c420456, 2026-10-25.

  • internal/datastore/migrations/migrations.go:40-43: lock.NewPostgresSessionLocker(lock.WithLockTimeout(1, 300), …).
  • internal/datastore/migrations/runner.go:28: upTimeout = 5 * time.Minute. At :172, _, err = r.up(upCtx, r.client.DB(), r.logger) discards the []*goose.MigrationResult, which carries a Duration per migration.
  • internal/datastore/postgres.go:45-56: the migration client is a separate postgres.Client with a pool of one connection. It has no pool collector. cmd/artifact-registry/wire_database.go:79-94 builds the runner and passes it no registerer.
  • cmd/artifact-registry/migration_runner.go:58-76: the pass runs in a goroutine while the server serves /-/metrics, so a gauge set on success is scrapeable. A failed pass calls onFatal and the process exits, so a failure-only series would rarely be scraped. The crash loop is already covered by KubeContainersWaitingInError (#1664 notes).

Fix direction

  1. Pass a prometheus.Registerer (the prefixed appReg) into migrations.NewRunner from wire_database.go.
  2. Publish:
    • database_migration_last_duration_seconds: a gauge with the wall time of the last successful Up, lock wait included. #1664's sketch already reads gitlab_artifact_registry_database_migration_last_duration_seconds.
    • database_migration_last_lock_wait_seconds: a gauge with the time spent in SessionLock, measured by a thin wrapper around goose's lock.SessionLocker.
    • database_migrations_applied_total: a counter of the migrations this process applied, so it is 0 on a no-op boot.
    • Optional: database_migration_last_success_timestamp_seconds. Seed every series at registration, so a no-op boot publishes 0 and not an absent series.
  3. In the same place, emit the #1380 (closed) structured completion record on every successful Up: duration, applied count and versions, and final state. Log around the r.up call from the returned []*goose.MigrationResult, or enable goose.WithVerbose.
  4. Update the S03-B ### Database table, docs/dev/observability.md, and S04's Advisory locking section if it describes observability. S04 owns the advisory-lock contract that runner.go:24-27 cites.

Acceptance criteria

  • After a successful startup pass, the three series show the pass's duration, its lock wait and the number of migrations applied. A no-op boot shows 0 applied and a small duration.
  • An integration test holds the advisory lock from another session, then releases it, and asserts that the lock-wait gauge is at least the held time.
  • Every successful Up emits one structured completion log record with duration, applied versions and final state (#1380 (closed)'s ask). A test asserts it.
  • The cardinality audit passes. The S03-B table and docs/dev/observability.md list the families.
  • #1664's MigrationNearBudget sketch is checked against the shipped names (a comment on #1664).

Size and fit

About 450-600 Go lines including the integration test and the log record. This fits one smurfit unit.

Dependencies

  • None blocking. Unblocks #1664 (MigrationNearBudget).
  • Related: #1410 (the table-size ceiling against upTimeout).
  • Related: #1569 (the Migrator switch) removes the boot-time runner (runner.go, migration_runner.go). If #1569 is scheduled before this issue starts, re-scope this issue to the Migrator Job before implementing it; the metrics and the log record would not survive the switch as written.
Edited by Pawel Rozlach