Publish transaction duration and outcome from the datastore transaction envelope
Split from #1606 (closed) (gap 1). Parent epic: &24088. Blocks #1664 (ArtifactRegistryDatabaseTransactionCommitFailures).
Decisions (operator ruling 2026-10-25)
- Q1, naming:
RunInTxandrunInSQLTxtake a required transaction name, and every caller (about 45) is named in this issue. There is no unnamed default and no parallelRunInNamedTx. - Q2, labels: the families use the new labels
transactionandtransaction_outcome. They do not reusename(budget 700, about 616 used) or the shared 26-valueoutcomeclosed set. - Q3, scope: the 27 direct
BeginTxsites are out of scope, except the two background transactions listed below. They are #1720 (closed), which depends on this issue. - Q7, spec: this MR amends S03-B itself. Land after #1718 (closed) where possible, which catches up the existing rows.
Problem
Named statements are timed. A transaction is not. runInSQLTx (internal/datastore/tx.go:75-120) and RunInTx (internal/datastore/tx.go:51-53) begin, commit and roll back a transaction and publish no duration, no commit failure and no rollback. A named lock statement measures the lock wait. Nothing measures the hold, which lasts until commit (#976, #944).
The four transactions that matter most for the audited areas:
| Transaction | Envelope | Where |
|---|---|---|
| S20 purge chunk (one reap call per chunk) | datastore.RunInTx |
internal/lifecycle/chunk.go:119 |
| S20 tombstone stamp | direct pool.BeginTx / tx.Commit |
internal/lifecycle/tombstone.go:116, :158 |
| S22 drain chunk (it also calls Redis inside the transaction) | datastore.RunInTx |
internal/accounting/chunk_worker.go:391 |
| S22 drain fan-out | direct BeginTx / tx.Commit |
internal/accounting/drain_trigger.go:255, :288 |
RunInTx / runInSQLTx have about 45 callers on origin/main (git grep -nE "RunInTx\(|runInSQLTx\(" -- '*.go' ':!*_test.go'). Another 27 sites call BeginTx directly. The direct sites other than the two above belong to #1720 (closed).
Evidence
Research base: origin/main at e2c420456, 2026-10-25.
internal/datastore/tx.go:76client.DB().BeginTx(ctx, nil),:114tx.Commit(),:103tx.Rollback(). Nometricsimport in the file.internal/metrics/database.go:37-67: the only database vectors are the fourname-labelled query vectors.- The cardinality audit counts the distinct values of a label across every family (
internal/metrics/audit.go:79-81).labelNamehas a budget of 700 (internal/metrics/cardinality.go:49). The two catalogs already declare about 616 names (603 ininternal/datastore/query_names.go, 13 ininternal/storage/queries.go). Putting the transaction names onnamewould use most of the headroom that is left. - The
outcomelabel is a shared closed set of 26 values (internal/metrics/cardinality.go:180,:510-518). The sibling labelcancellation_reasonwas given its own name for this reason (internal/metrics/metrics.go:28).
Fix direction
- In
internal/metrics, add:database_transaction_duration_seconds{transaction}: a histogram fromBeginTxto the end ofCommitorRollback. Use its own bucket family that reaches past the statement ceiling, for example0.005 … 5, 10, 30, 60, 300. A purge chunk or a drain chunk can hold its transaction longer than any one statement.database_transaction_outcomes_total{transaction, transaction_outcome}, withtransaction_outcome ∈ {committed, commit_error, rolled_back, panic}pinned inclosedSetValues.rolled_backis a rollback thatfncaused.commit_erroris aCommitthat failed.- An
InstrumentTx(name) func(*error)closure helper, in the same form asInstrumentQueryErr(internal/metrics/database.go:120-137). It must be deferred directly sorecoverworks. - An
InitTransaction(name)seeding hook, called at startup next tometrics.InitQuery(cmd/artifact-registry/main.go:307).
- Add a required
nameparameter torunInSQLTxandRunInTx, and name every caller. Declare the names in a transaction catalog ininternal/datastore, for exampletx_names.go, as<area>_<verb>_tx. - Move the tombstone transaction and the drain fan-out onto the helper. Either route them through
RunInTx, or deferInstrumentTxat theirBeginTx. - Optional: give the purge chunk one name per reaper format (
lifecycle_purge_chunk_{maven,npm,container}_tx) instead of one shared name. That gives #976 the npm reap hold time. - Update
docs/dev/observability.md(two rows),docs/dev/database-query-patterns.md## Transactions, the S03-B### Databasetable (docs/specs/S03-b-metrics.md:348-353), the bucket-family table (:458-462) and the expected-distinct-values table (:535). Add thetransactionandtransaction_outcomebudgets toexpectedDistinctValues.
Acceptance criteria
-
database_transaction_duration_secondsanddatabase_transaction_outcomes_totalare published for everyRunInTxandrunInSQLTxcall, each under a static catalog name. The name parameter is required. - The S20 purge chunk, the S20 tombstone, the S22 drain chunk and the S22 drain fan-out each appear under their own
transactionvalue. - A failing
Commitbookscommit_error, anfnerror booksrolled_back, and a panic bookspanicand re-panics. Each case has a unit test. - Every catalog name is seeded at startup, so
increase()sees the firstcommit_error. -
TestEveryStatementIsInstrumented, or a sibling test, fails on aRunInTx/runInSQLTxcall whose name is not in the catalog, or whose name is used twice. - The cardinality audit passes. The S03-B tables,
observability.mdanddatabase-query-patterns.mdmatch the code. - #1664's
TransactionCommitFailuressketch is updated to the shipped metric and label names (transaction_outcome="commit_error", notoutcome=). A comment on #1664 is enough.
Size and fit
About 600-900 Go lines: the metrics helper and tests are about 250, the catalog and the call-site edits are about 45 × 2-3 lines, and the sweep test is about 100. This fits one smurfit unit because the 27 direct BeginTx sites stay in #1720 (closed).
Dependencies
- None blocking. It touches
internal/metrics/metrics.goand the S03-B bucket table, the same as #1723 (closed), so rebase whichever lands second. - Related: #590 and #589 also change the
runInSQLTxsignature (a*sql.TxOptionsvariant). Whichever lands second rebases onto the other. - Unblocks #1664 (
TransactionCommitFailures) and #1720 (closed).