feat(metrics): book raw-SQL site outcomes under their query names

Summary

Problem. The outcome counters of a named query, database_query_errors_total{name} and database_query_cancellations_total{name,cancellation_reason}, are booked by metrics.InstrumentQueryErr. The 30 production sites that run raw SQL called the plain metrics.InstrumentQuery timer instead, and that timer books duration and count only. Each of those names therefore exported both outcome series at 0 for its whole life. A name that never failed and a name that books no outcome at all looked the same.

Impact. The errors / queries ratio of those names read 0 while the statement failed. The database alert rules tracked in #1664 depend on these series.

Change. Every one of the 30 sites takes done := metrics.InstrumentQueryErr(<name>) and then defer done(&err) over the function's own named error result, at the site's existing placement. The split is 29 sites in internal/datastore and 1 in internal/storage, counted at head 6ed4fc4e2. Two source-reading tests hold the rule in the two packages. The plain timer must not appear in a non-test file. Every pinned raw-SQL function must defer the error-aware closure over its own named error result. The deferred-timer exception leaves the four docs/dev/ pages and the two code comments that stated it.

Review fixes. Four commits after the feature commit answer review findings:

  • The blob-reference probe read in internal/datastore/repo_blob_references.go returns rows.Err() alone when it is set, and returns sql.ErrNoRows only for a clean empty set. Before, the two errors were joined, so the timer booked a cancellation or a driver fault on the probe read as no row.
  • The doc comment of TestEveryStatementIsTimed opens again with the invariant that the test asserts.
  • docs/dev/database-query-patterns.md drops the last qualifier that was left from the deferred-timer exception.
  • docs/dev/storage-accounting.md gives the reason that a failed reap still needs its own signal: a guard refusal before the timer starts books no metric.

Downsides. The per-site proof is a source sweep, not a failure-path test: the runtime pass did not drive the failure and cancellation arm. The same AST helper functions appear in both test files, and the branch leaves them there: cross-package duplication is legal, and a shared test-helper package moves declarations out of a shared-seam file. metrics.InstrumentQuery stays exported and unchanged, because the spec describes it, so the plain timer stays callable.

Checklist

  • Ran it: The phase-3 runtime pass booted the branch with the run-artifact-registry driver and ran its smoke flow: build, boot, seed of npm, Maven and OCI repositories, and 103 HTTP assertions driven with curl. All passed. It then scraped /-/metrics. upload_sessions_commit_tx reads queries_total 10 with both outcome series present at 0, and npm_versions_update_package_json_batch, which the smoke never calls, is present and zeroed. That is the live confirmation that the zero-floor registration now covers these raw-SQL names.
  • Interfaces and data: None. No API, configuration, schema, or wire or storage format changes. The one observable surface is the /-/metrics exposition: the 30 names now book outcome series. metrics.InstrumentQuery stays exported and unchanged, and the duration and count contract of S03-B holds. No docs/user/ page is affected: /-/metrics is operator-facing.
  • Failure: A converted site books one database_query_errors_total{name} when its statement fails, and one database_query_cancellations_total{name,cancellation_reason} when the caller's context ends first. A nil error and a no-row sentinel book neither. A panic inside the timed span books one error and still propagates. The values and the errors that the callers receive do not change, so a client sees no difference.
  • Rollout: No migration and no configuration change, so the previous release runs against the same schema and configuration during a deploy. The new series appear with this release.
  • Hot path: None. No query, allocation, or unbounded work is added: each site swaps one deferred closure for another of the same shape at the same place.
  • Database: No. No migration is added or changed, and no SQL text changes. The diff replaces the timer that wraps each existing statement.
  • Security: None. No new input, route, permission, or audited action, and no change to how a caller is authenticated or authorized.
  • ADRs: None governs it. Checked: ADR-023 (code structure) — no package and no exported symbol is added; ADR-012 (usage data collection) — the operational Prometheus metrics are outside its decision; ADR-006 (technology stack) — no new dependency. No ADR covers query-metric naming or the outcome counters.
  • E2E: No scenario row is added. The change asserts a scrape-level metric, not a client journey, and no Maven, npm, or OCI protocol behavior changes, so no row is affected.
  • Simplicity: The card left the site form open: a named error result with defer done(&err) at each site, or a raw-SQL helper in internal/datastore/queries.go. The branch took the first, and the committed sweep rejects the plain timer, so the helper has no use. InstrumentQuery was left exported, because S03-B describes it.

Acceptance coverage

Rows from the card's derived.acceptance A1–A8; spec: none governs.

# Item Tests
A1 No metrics.InstrumentQuery( call outside internal/metrics TestEveryStatementIsInstrumented and TestEveryStatementIsTimed reject a plain-timer call in any non-test file of internal/datastore and internal/storage; the repository-wide grep is the review-time check (see Gaps)
A2 A statement error that is neither a no-row sentinel nor a context error increments database_query_errors_total{name} by 1 for the name the call ran under defersErrorAwareTimer proves each converted function hands its own named error result to the closure; the booking is pinned on that shared closure by TestInstrumentQuery_ObservesUnderNameAndReturnsError, TestInstrumentExec_CountsAFailedStatement and TestInstrumentQuery_BooksTheOutcomeOfFn
A3 context.Canceled increments database_query_cancellations_total{name,cancellation_reason="client"}, context.DeadlineExceeded {cancellation_reason="deadline"}, and neither moves database_query_errors_total{name} TestInstrumentQuery_BooksTheOutcomeOfFn ("cancelled", "deadline")
A4 A nil error, qrm.ErrNoRows or sql.ErrNoRows books on neither outcome counter TestInstrumentQuery_AbsentRowIsNoFailure, TestInstrumentQuery_BooksTheOutcomeOfFn ("success", "absent row", "absent sql row")
A5 A panic inside a converted function's timed span books one database_query_errors_total{name} and still propagates TestInstrumentQuery_PanicBooksAFailure
A6 Each converted call still observes database_query_duration_seconds{name} and increments database_queries_total{name} once TestInstrumentQuery_ObservesUnderNameAndReturnsError, TestInstrumentExec_ThreadsResultAndCounts, TestInstrumentQuery_CoversS06NameSet
A7 The source sweeps fail on a function that runs a raw statement under the plain timer TestEveryStatementIsInstrumented (internal/datastore) and TestEveryStatementIsTimed (internal/storage) fail when a pinned function does not defer the error-aware closure over its own named error result, and when a non-test file holds a plain-timer call
A8 The deferred-timer exception is removed from docs/dev/observability.md, docs/dev/database-query-patterns.md, internal/metrics/database.go and the files-listed spots No Go test: the row is documentation and comments. The branch removes the exception from docs/dev/database-query-patterns.md, docs/dev/observability.md, docs/dev/storage.md and docs/dev/storage-accounting.md, and from the code comments in internal/metrics/database.go and internal/datastore/queries.go

Gaps the rows name: A1's repository-wide grep and A8 carry no test in the two test files, and no test drives a converted site's failure path (see the Downsides answer above).

Merge order

!2944 (closed) and !3470 (merged) change lines that this branch also changes. !2944 (closed) rewrites TestEveryStatementIsTimed in internal/storage/queries_test.go and the raw-site section of docs/dev/database-query-patterns.md, and it touches internal/datastore/container_blob.go and internal/storage/pg_session.go. !3470 (merged) edits the startup rule in docs/dev/observability.md. Whichever of the two lands first, this branch rebases over it.

Related to #1208 (closed) What remains: the acceptance items are the enricher's derivation (the card's derived.acceptance A1–A8) rather than an outcome the issue states, so the Closes condition is not met and the issue stays open for its author.

This is a bot message 🤖 — /smurfit

Edited by Pawel Rozlach

Merge request reports

Loading
Loading