Publish transaction duration and outcome from the datastore transaction envelope

Split from #1606 (closed) (gap 1). Parent epic: &24088. Blocks #1664 (ArtifactRegistryDatabaseTransactionCommitFailures).

Decisions (operator ruling 2026-10-25)

  • Q1, naming: RunInTx and runInSQLTx take a required transaction name, and every caller (about 45) is named in this issue. There is no unnamed default and no parallel RunInNamedTx.
  • Q2, labels: the families use the new labels transaction and transaction_outcome. They do not reuse name (budget 700, about 616 used) or the shared 26-value outcome closed set.
  • Q3, scope: the 27 direct BeginTx sites are out of scope, except the two background transactions listed below. They are #1720 (closed), which depends on this issue.
  • Q7, spec: this MR amends S03-B itself. Land after #1718 (closed) where possible, which catches up the existing rows.

Problem

Named statements are timed. A transaction is not. runInSQLTx (internal/datastore/tx.go:75-120) and RunInTx (internal/datastore/tx.go:51-53) begin, commit and roll back a transaction and publish no duration, no commit failure and no rollback. A named lock statement measures the lock wait. Nothing measures the hold, which lasts until commit (#976, #944).

The four transactions that matter most for the audited areas:

Transaction Envelope Where
S20 purge chunk (one reap call per chunk) datastore.RunInTx internal/lifecycle/chunk.go:119
S20 tombstone stamp direct pool.BeginTx / tx.Commit internal/lifecycle/tombstone.go:116, :158
S22 drain chunk (it also calls Redis inside the transaction) datastore.RunInTx internal/accounting/chunk_worker.go:391
S22 drain fan-out direct BeginTx / tx.Commit internal/accounting/drain_trigger.go:255, :288

RunInTx / runInSQLTx have about 45 callers on origin/main (git grep -nE "RunInTx\(|runInSQLTx\(" -- '*.go' ':!*_test.go'). Another 27 sites call BeginTx directly. The direct sites other than the two above belong to #1720 (closed).

Evidence

Research base: origin/main at e2c420456, 2026-10-25.

  • internal/datastore/tx.go:76 client.DB().BeginTx(ctx, nil), :114 tx.Commit(), :103 tx.Rollback(). No metrics import in the file.
  • internal/metrics/database.go:37-67: the only database vectors are the four name-labelled query vectors.
  • The cardinality audit counts the distinct values of a label across every family (internal/metrics/audit.go:79-81). labelName has a budget of 700 (internal/metrics/cardinality.go:49). The two catalogs already declare about 616 names (603 in internal/datastore/query_names.go, 13 in internal/storage/queries.go). Putting the transaction names on name would use most of the headroom that is left.
  • The outcome label is a shared closed set of 26 values (internal/metrics/cardinality.go:180, :510-518). The sibling label cancellation_reason was given its own name for this reason (internal/metrics/metrics.go:28).

Fix direction

  1. In internal/metrics, add:
    • database_transaction_duration_seconds{transaction}: a histogram from BeginTx to the end of Commit or Rollback. Use its own bucket family that reaches past the statement ceiling, for example 0.005 … 5, 10, 30, 60, 300. A purge chunk or a drain chunk can hold its transaction longer than any one statement.
    • database_transaction_outcomes_total{transaction, transaction_outcome}, with transaction_outcome ∈ {committed, commit_error, rolled_back, panic} pinned in closedSetValues. rolled_back is a rollback that fn caused. commit_error is a Commit that failed.
    • An InstrumentTx(name) func(*error) closure helper, in the same form as InstrumentQueryErr (internal/metrics/database.go:120-137). It must be deferred directly so recover works.
    • An InitTransaction(name) seeding hook, called at startup next to metrics.InitQuery (cmd/artifact-registry/main.go:307).
  2. Add a required name parameter to runInSQLTx and RunInTx, and name every caller. Declare the names in a transaction catalog in internal/datastore, for example tx_names.go, as <area>_<verb>_tx.
  3. Move the tombstone transaction and the drain fan-out onto the helper. Either route them through RunInTx, or defer InstrumentTx at their BeginTx.
  4. Optional: give the purge chunk one name per reaper format (lifecycle_purge_chunk_{maven,npm,container}_tx) instead of one shared name. That gives #976 the npm reap hold time.
  5. Update docs/dev/observability.md (two rows), docs/dev/database-query-patterns.md ## Transactions, the S03-B ### Database table (docs/specs/S03-b-metrics.md:348-353), the bucket-family table (:458-462) and the expected-distinct-values table (:535). Add the transaction and transaction_outcome budgets to expectedDistinctValues.

Acceptance criteria

  • database_transaction_duration_seconds and database_transaction_outcomes_total are published for every RunInTx and runInSQLTx call, each under a static catalog name. The name parameter is required.
  • The S20 purge chunk, the S20 tombstone, the S22 drain chunk and the S22 drain fan-out each appear under their own transaction value.
  • A failing Commit books commit_error, an fn error books rolled_back, and a panic books panic and re-panics. Each case has a unit test.
  • Every catalog name is seeded at startup, so increase() sees the first commit_error.
  • TestEveryStatementIsInstrumented, or a sibling test, fails on a RunInTx / runInSQLTx call whose name is not in the catalog, or whose name is used twice.
  • The cardinality audit passes. The S03-B tables, observability.md and database-query-patterns.md match the code.
  • #1664's TransactionCommitFailures sketch is updated to the shipped metric and label names (transaction_outcome="commit_error", not outcome=). A comment on #1664 is enough.

Size and fit

About 600-900 Go lines: the metrics helper and tests are about 250, the catalog and the call-site edits are about 45 × 2-3 lines, and the sweep test is about 100. This fits one smurfit unit because the 27 direct BeginTx sites stay in #1720 (closed).

Dependencies

  • None blocking. It touches internal/metrics/metrics.go and the S03-B bucket table, the same as #1723 (closed), so rebase whichever lands second.
  • Related: #590 and #589 also change the runInSQLTx signature (a *sql.TxOptions variant). Whichever lands second rebases onto the other.
  • Unblocks #1664 (TransactionCommitFailures) and #1720 (closed).
Edited by Pawel Rozlach