feat(metrics): book raw-SQL site outcomes under their query names
Summary
Problem. The outcome counters of a named query, database_query_errors_total{name} and
database_query_cancellations_total{name,cancellation_reason}, are booked by
metrics.InstrumentQueryErr. The 30 production sites that run raw SQL called the plain
metrics.InstrumentQuery timer instead, and that timer books duration and count only. Each
of those names therefore exported both outcome series at 0 for its whole life. A name that
never failed and a name that books no outcome at all looked the same.
Impact. The errors / queries ratio of those names read 0 while the statement failed.
The database alert rules tracked in #1664 depend on these series.
Change. Every one of the 30 sites takes done := metrics.InstrumentQueryErr(<name>) and
then defer done(&err) over the function's own named error result, at the site's existing
placement. The split is 29 sites in internal/datastore and 1 in internal/storage,
counted at head 6ed4fc4e2. Two source-reading tests hold the rule in the two packages. The
plain timer must not appear in a non-test file. Every pinned raw-SQL function must defer the
error-aware closure over its own named error result. The deferred-timer exception leaves the
four docs/dev/ pages and the two code comments that stated it.
Review fixes. Four commits after the feature commit answer review findings:
- The blob-reference probe read in
internal/datastore/repo_blob_references.goreturnsrows.Err()alone when it is set, and returnssql.ErrNoRowsonly for a clean empty set. Before, the two errors were joined, so the timer booked a cancellation or a driver fault on the probe read as no row. - The doc comment of
TestEveryStatementIsTimedopens again with the invariant that the test asserts. docs/dev/database-query-patterns.mddrops the last qualifier that was left from the deferred-timer exception.docs/dev/storage-accounting.mdgives the reason that a failed reap still needs its own signal: a guard refusal before the timer starts books no metric.
Downsides. The per-site proof is a source sweep, not a failure-path test: the runtime
pass did not drive the failure and cancellation arm. The same AST helper functions appear in
both test files, and the branch leaves them there: cross-package duplication is legal, and a
shared test-helper package moves declarations out of a shared-seam file.
metrics.InstrumentQuery stays exported and unchanged, because the spec describes it, so the
plain timer stays callable.
Checklist
- Ran it: The phase-3 runtime pass booted the branch with the
run-artifact-registrydriver and ran its smoke flow: build, boot, seed of npm, Maven and OCI repositories, and 103 HTTP assertions driven withcurl. All passed. It then scraped/-/metrics.upload_sessions_commit_txreadsqueries_total10 with both outcome series present at0, andnpm_versions_update_package_json_batch, which the smoke never calls, is present and zeroed. That is the live confirmation that the zero-floor registration now covers these raw-SQL names. - Interfaces and data: None. No API, configuration, schema, or wire or storage format
changes. The one observable surface is the
/-/metricsexposition: the 30 names now book outcome series.metrics.InstrumentQuerystays exported and unchanged, and the duration and count contract of S03-B holds. Nodocs/user/page is affected:/-/metricsis operator-facing. - Failure: A converted site books one
database_query_errors_total{name}when its statement fails, and onedatabase_query_cancellations_total{name,cancellation_reason}when the caller's context ends first. A nil error and a no-row sentinel book neither. A panic inside the timed span books one error and still propagates. The values and the errors that the callers receive do not change, so a client sees no difference. - Rollout: No migration and no configuration change, so the previous release runs against the same schema and configuration during a deploy. The new series appear with this release.
- Hot path: None. No query, allocation, or unbounded work is added: each site swaps one deferred closure for another of the same shape at the same place.
- Database: No. No migration is added or changed, and no SQL text changes. The diff replaces the timer that wraps each existing statement.
- Security: None. No new input, route, permission, or audited action, and no change to how a caller is authenticated or authorized.
- ADRs: None governs it. Checked: ADR-023 (code structure) — no package and no exported symbol is added; ADR-012 (usage data collection) — the operational Prometheus metrics are outside its decision; ADR-006 (technology stack) — no new dependency. No ADR covers query-metric naming or the outcome counters.
- E2E: No scenario row is added. The change asserts a scrape-level metric, not a client journey, and no Maven, npm, or OCI protocol behavior changes, so no row is affected.
- Simplicity: The card left the site form open: a named error result with
defer done(&err)at each site, or a raw-SQL helper ininternal/datastore/queries.go. The branch took the first, and the committed sweep rejects the plain timer, so the helper has no use.InstrumentQuerywas left exported, because S03-B describes it.
Acceptance coverage
Rows from the card's derived.acceptance A1–A8; spec: none governs.
| # | Item | Tests |
|---|---|---|
| A1 | No metrics.InstrumentQuery( call outside internal/metrics |
TestEveryStatementIsInstrumented and TestEveryStatementIsTimed reject a plain-timer call in any non-test file of internal/datastore and internal/storage; the repository-wide grep is the review-time check (see Gaps) |
| A2 | A statement error that is neither a no-row sentinel nor a context error increments database_query_errors_total{name} by 1 for the name the call ran under | defersErrorAwareTimer proves each converted function hands its own named error result to the closure; the booking is pinned on that shared closure by TestInstrumentQuery_ObservesUnderNameAndReturnsError, TestInstrumentExec_CountsAFailedStatement and TestInstrumentQuery_BooksTheOutcomeOfFn |
| A3 | context.Canceled increments database_query_cancellations_total{name,cancellation_reason="client"}, context.DeadlineExceeded {cancellation_reason="deadline"}, and neither moves database_query_errors_total{name} | TestInstrumentQuery_BooksTheOutcomeOfFn ("cancelled", "deadline") |
| A4 | A nil error, qrm.ErrNoRows or sql.ErrNoRows books on neither outcome counter | TestInstrumentQuery_AbsentRowIsNoFailure, TestInstrumentQuery_BooksTheOutcomeOfFn ("success", "absent row", "absent sql row") |
| A5 | A panic inside a converted function's timed span books one database_query_errors_total{name} and still propagates | TestInstrumentQuery_PanicBooksAFailure |
| A6 | Each converted call still observes database_query_duration_seconds{name} and increments database_queries_total{name} once | TestInstrumentQuery_ObservesUnderNameAndReturnsError, TestInstrumentExec_ThreadsResultAndCounts, TestInstrumentQuery_CoversS06NameSet |
| A7 | The source sweeps fail on a function that runs a raw statement under the plain timer | TestEveryStatementIsInstrumented (internal/datastore) and TestEveryStatementIsTimed (internal/storage) fail when a pinned function does not defer the error-aware closure over its own named error result, and when a non-test file holds a plain-timer call |
| A8 | The deferred-timer exception is removed from docs/dev/observability.md, docs/dev/database-query-patterns.md, internal/metrics/database.go and the files-listed spots | No Go test: the row is documentation and comments. The branch removes the exception from docs/dev/database-query-patterns.md, docs/dev/observability.md, docs/dev/storage.md and docs/dev/storage-accounting.md, and from the code comments in internal/metrics/database.go and internal/datastore/queries.go |
Gaps the rows name: A1's repository-wide grep and A8 carry no test in the two test files, and no test drives a converted site's failure path (see the Downsides answer above).
Merge order
!2944 (closed) and !3470 (merged) change lines that this branch also changes. !2944 (closed) rewrites
TestEveryStatementIsTimed in internal/storage/queries_test.go and the raw-site section of
docs/dev/database-query-patterns.md, and it touches internal/datastore/container_blob.go
and internal/storage/pg_session.go. !3470 (merged) edits the startup rule in
docs/dev/observability.md. Whichever of the two lands first, this branch rebases over it.
Related to #1208 (closed)
What remains: the acceptance items are the enricher's derivation (the card's derived.acceptance A1–A8) rather than an outcome the issue states, so the Closes condition is not met and the issue stays open for its author.
This is a bot message