feat(metrics): bound the database query histogram past five seconds
Summary
Problem. dbLatencyBuckets ended at a finite 5 s bucket. The background statements that run on the same pool are slower than that:
- a reap page,
- a reconciliation sum,
- a drain lock wait,
- a tombstone count,
- the tombstone stamp under its 10 s budget.
Every observation past 5 s landed in +Inf. histogram_quantile reads the top finite boundary for a sample in +Inf. So a 7 s statement and a 25 s one both read as 5 s.
Impact. The database latency panels cannot separate those statements. An operator who reads a p99 for a named query sees the old ceiling, not the statement's duration. No client sees a difference.
Change. The family gains finite 10 s and 30 s bounds, so it is 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30. The 10 s bound brackets the tombstone stamp's 10 s budget and the reconciliation sum's 5 s budget. The 30 s bound gives a statement past 10 s a bucket of its own. A reap page or a drain lock wait has no per-statement budget, only the 30-minute deadline of its purge attempt. So one of them that runs past 30 s still lands in +Inf. S03-B's bucket table and its rationale paragraph move with it. So do the observability catalog rows, the comments in the four packages that serve the family, and the three Markdown pages beside that code. reconcileStatementTimeout keeps its 5 s value, and its comment no longer gives the bucket ceiling as a reason for it. Two new test cases pin the family and the placement, and the purge-lag guard reads the shared family's top instead of a literal 5 s.
Downsides. Every series of the shared family gains two buckets, and so do the two histograms that read it: remote_negative_cache_duration_seconds and maven_virtual_merge_duration_seconds. A quantile below 5 s does not move. One above it now reads a value above 5 s instead of exactly 5 s, which is the point of the change.
Checklist
- Ran it: Nothing yet against a running service. No client drives this change. The only runtime surface is a scrape of
/-/metrics. That scrape must showle="10"andle="30"beside the unchangedle="5"ondatabase_query_duration_seconds, and on the two histograms that share the family. What ran instead: the package tests and the repository's gate battery (gate-run --changed --base origin/main), where build, vet, lint, gofmt, tests and the docs checks pass. The battery'slint-integrationrow reports the repository's pre-existing 5875-finding baseline, which this branch does not change. - Interfaces and data: The metric exposition changes, and nothing else.
database_query_duration_secondsgainsle="10"andle="30", and so do the two histograms that readmetrics.DBLatencyBuckets. No API, configuration, schema, wire or storage format changes. Existing clients and stored data are unaffected. The family keeps its name, its labels and every boundary below 5 s, and the two new bounds sit at the top.docs/specs/S03-b-metrics.mdstates the new family in this merge request.docs/user/does not document these families, so no user documentation is affected. - Failure: n/a: the change adds no error path, no timeout and no write. It has no new failure mode, and no client sees a difference. A scrape grows by two
leseries per labeled series. The cardinality audit reads labels rather than bucket counts, so it cannot fail on the new bounds. - Rollout: n/a for the schema and the configuration: neither changes. During a rolling deploy, a pod on the old binary exports eleven buckets and a pod on the new one exports thirteen. The
le="10"andle="30"series start at each new pod's first scrape. A quantile over a window that spans the deploy therefore reads those two buckets from partial data for that window. A rollback drops the two series again and leaves the eleven. - Hot path: No.
dbLatencyBucketsis a package-level slice that the process builds once. The observation path is unchanged apart from one more comparison in its bucket search. No query and no unbounded work reaches a request path. - Database: n/a: the change adds no migration and no change to a query. It moves the buckets of the histogram that times queries. The queries themselves are untouched, so the database review evidence does not apply.
- Security: No. The change adds no input, route, permission or audited action, and it changes no authentication or authorization. The family keeps its single
namelabel, and a bucket boundary is not a label, so no new value can reach a label./-/metricsstays on the probe listener, unauthenticated and internal-only. - ADRs: None governs it.
docs/specs/S03-b-metrics.mdpins the family, and this merge request amends it. The ADR set was searched for metric, latency, percentile and bucket constraints, and none holds one for the database-query family. - E2E: No scenario row is added or changed. The journey is observable from outside: scrape
/-/metricsafter a statement past 5 s and read the two buckets. A row would need a statement deliberately held past 5 s, and the boundaries are already pinned by the unit tests that the spec's AC-5 and AC-8 rows rest on. The existing rowe2e.management.observability.metric-families-registeredis unaffected: this change registers no family and removes none. - Simplicity: The alternative was to give the two histograms that share
DBLatencyBucketstheir own copy of the old bounds, so that only the database family would widen. The decision is that they keep sharing the family and gain the two bounds with it. One bound would not be enough: the issue needs a 7 s and a 25 s statement in different buckets, and that takes a boundary between them and a top above the 25 s one.
Spec coverage
Spec: docs/specs/S03-b-metrics.md. Its acceptance criteria are framework-level
and name no bucket family, so every one of them is listed below with its
disposition, followed by the section this MR amends; the card's acceptance
items for issue #1723 (closed) are the binding rows.
Acceptance criteria (S03-B)
| # | Criterion | Tests |
|---|---|---|
| AC-1 | Every AR application metric resolves to gitlab_artifact_registry_* through the prefixed registerer |
Not touched: two boundaries are added to an existing family; no name, prefix or registration changes. Existing cases stay green. |
| AC-2 | The HTTP server middleware records the matched route pattern | Not touched: the HTTP families are untouched. |
| AC-3 | Panic accounting on a recovered panic | Not touched. |
| AC-4 | http_in_flight_requests carries only method |
Not touched. |
| AC-5 | The named-query instrumenter records database_query_duration_seconds{name} and database_queries_total{name}, inert until registration |
Not touched: the instrumenter does not change. Existing TestInstrumentQuery_RecordsBothSeries and TestInstrumentQuery_InertBeforeRegistration stay green. |
| AC-6 | The pool collector reads live pool stats at scrape time | Not touched. |
| AC-7 | Latency histograms record a trace_id exemplar under a span |
Not touched: the DB instrumenter records no exemplar by design (spec ### Exemplars). |
| AC-8 | The cardinality audit: disallowed-name, coverage, closed-set and count-bound checks | The audit cannot fail on bucket count (F4): it reads labels, and a boundary is not a label. Existing TestAuditCardinality_*, TestClosedSetValues_* and TestObservePurge_PassesTheCardinalityAudit stay green; no new test. |
| AC-9 | OpenMetrics with exemplars on the #58 listener | Gated on #58; not touched. |
| AC-10 | database_connection_pool_max_size publishes no series without a pool |
Not touched. |
| AC-11 | An invalid-UTF-8 r.Pattern records route="" |
Not touched. |
S03-B ### Histogram buckets (the section this MR amends) |
The DB query latency family is pinned at 0.001 … 5; this MR appends 10, 30 and restates the rationale |
TestDBLatencyBuckets_CarryTheBackgroundStatementBounds (new pin), TestDatabaseQueryDuration_SeparatesStatementsPastFiveSeconds (new: observes a 7 s and a 25 s statement directly on the database-query family, not through the instrumenter), TestBucketFamilies (existing strictly-increasing guard) |
Acceptance criteria (card acceptance, issue #1723 (closed))
| # | Acceptance item | Tests |
|---|---|---|
| A1 | A statement that takes 7 s and one that takes 25 s land in different buckets (le="10" and le="30"), and a histogram_quantile over them reads above 5 s |
TestDatabaseQueryDuration_SeparatesStatementsPastFiveSeconds (cumulative counts at le="5", le="10", le="30"; the quantile follows from the placement, F7), TestDBLatencyBuckets_CarryTheBackgroundStatementBounds (the bounds exist at all) |
| A2 | S03-B's bucket table and its rationale paragraph state the new bounds; docs/dev/observability.md and docs/dev/database-query-patterns.md agree |
No test: prose. The implementation commit carries the doc edits, and nothing in the repository asserts doc text against the code. |
| A3 | Every listed comment describes the new ceiling; reconcileStatementTimeout keeps its value and its comment no longer gives the bucket ceiling as a reason |
Value half: existing internal/accounting/reconcile_task_internal_test.go pins the scan's deadline at reconcileStatementTimeout, unchanged and green. Comment half is prose and has no test. |
| A4 | The #944 bucket question is answered on #944 | Owned by a dependency: the run posts the note when the MR opens (card acceptance_4, F3). No test row. |
| A5 | The #908 (closed) series-per-name figure is updated, and the cardinality audit passes | Audit half: the existing cardinality tests above, which cannot fail on bucket count (F4). The #908 (closed) figure is a note (card acceptance_5), not a test. |
Error cases
| Condition | Tests |
|---|---|
| A collector is registered twice on the same registry | Not touched: no collector is added or registered differently. Existing duplicate-registration cases stay green. |
| A label value outside its declared allowlist is observed | Not touched: the change adds no label and no label value. |
| Code registers on the prometheus global default registry | Not touched. |
Accept: application/openmetrics-text before the #58 listener lands |
Not touched. |
A histogram receives a negative or NaN observation |
Not touched: the change widens a bucket family; it feeds no value. |
r.Pattern is not valid UTF-8 at the middleware exit |
Not touched. |
Security considerations
| Consideration | Tests |
|---|---|
| No high-cardinality or sensitive values in metric labels | Not touched: a bucket boundary is not a label; the family keeps its one name label, already classified in the expected-distinct-values table, and the audit's descriptor check stays green. |
/-/metrics is unauthenticated and internal-only |
Not touched. |
| No secrets in metric names or help text | Not touched: no name and no help string changes. |
Size at bbe145ef0: 12 files, +111 / -35 — 15 added lines in production Go, 76 in Go tests, 11 under docs/, and 9 in Markdown beside the code.
Merge order
- Merge this after #1718 (closed) ("Bring the S03-B database metrics table up to the shipped families"). The issue's Decision Q7 says #1718 (closed) lands first, and #1718 (closed) is open. Its Dependencies section says "None blocking", and the link list holds no blocker either. The two merge requests' S03-B hunks do not overlap.
- Eleven open sibling merge requests carry "the database family stops at 5 s" prose, which this branch makes stale: !3460 (merged), !3468 (closed), !3470 (merged), !3473 (closed), !3475 (merged), !3478 (merged), !3480 (merged), !3487 (merged), !3523 (merged), !3524 (merged) and !3525 (merged). Whoever lands second reconciles them, because this branch cannot edit another merge request's text.
- !3473 (closed) also overlaps
docs/dev/observability.md: it edits the samelifecycle_purge_*rows this branch substitutes "the database-query family at 5" → "at 30" in. The overlap is recorded rather than resolved here, because a splice would carry an unmerged merge request's prose into this diff. It is resolved at rebase time, against !3473 (closed)'s landed text. docs/plans/2026-08-11-s20a-lifecycle-closed-beta.md(Step 4 and Step 12) still states the old 5 s ceiling. This branch is not a step merge request of that plan, so it carries no plan correction for it.
Related to #1723 (closed)
What remains outside this merge request: the note on #944 that answers its bucket question, and the note on #908 (closed) that updates its series-per-name figure. #944's other half, "a parked statement exports nothing", stays open there.
This is a bot message