Let the database query histogram tell a background statement past 5 s apart from one at 5 s
Split from #1606 (closed) (gap 3). Parent epic: &24088. Settles the bucket half of #944.
Decision (operator ruling 2026-10-25)
- Q5, buckets: append
10, 30todbLatencyBuckets, so the family becomes0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30. This costs about 1,230 series per pod (2 bounds × about 616 names). It covers the longest production statement budget (TombstoneTimeout = 10 s) with headroom, in one family, with no catalog split. Transactions get their own wide family in #1719 (closed). - Q7, spec: this MR amends S03-B itself (the bucket family is spec-pinned at
docs/specs/S03-b-metrics.md:461). #1718 (closed) lands first.
Problem
dbLatencyBuckets ends at 5 s (internal/metrics/metrics.go:41), and S03-B pins that family (docs/specs/S03-b-metrics.md:461). The following all land in +Inf, so histogram_quantile reads exactly 5 s for them: reap pages, reconciliation sums, drain lock waits, tombstone counts and the tombstone stamp under its 10 s budget (#944). The #1607 dashboard has to describe "a flat 5 s line means over 5 s".
Evidence
Research base: origin/main at e2c420456, 2026-10-25.
internal/metrics/metrics.go:41:dbLatencyBuckets = []float64{0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5}. All four query vectors use it (internal/metrics/database.go:42), and so does the remote negative-cache histogram throughDBLatencyBuckets(:46).- Background statement budgets above or at 5 s:
internal/lifecycle/tombstone.go:57(TombstoneTimeout = 10 * time.Second),internal/accounting/reconcile_task.go:97(reconcileStatementTimeout = 5 * time.Second),internal/accounting/backlog_collector.go:16,internal/accounting/reconcile_trigger.go:232. - Several comments tie a value or a claim to the 5 s ceiling. They must change in the same MR (guardrail 16):
internal/accounting/reconcile_task.go:91-95("Five seconds is also the largest bound the timing can still tell apart");internal/accounting/reconcile_trigger.md:48;internal/datastore/lifecycle_reap_maven.md:88,:96;internal/datastore/reconcile_namespace.go:60-61;internal/datastore/reconcile_repository.go:457.
- Series cost: each name costs 15 series today, 12 of them
_bucket(#908 (closed)). Each added bound costs about 616 series per pod. - The remote negative-cache histogram shares
DBLatencyBuckets(internal/metrics/database.go:46), so it gains the two bounds too. Check that this is wanted, or give it its own copy of the old family.
Fix direction
- Append
10, 30todbLatencyBucketsininternal/metrics/metrics.go. - Update the S03-B bucket table and its rationale paragraph (
docs/specs/S03-b-metrics.md:458-471),docs/dev/observability.mdanddocs/dev/database-query-patterns.md. - Rewrite every comment listed above to describe the new ceiling.
Acceptance criteria
- A statement that takes 7 s and one that takes 25 s land in different buckets (
le="10"andle="30"), and ahistogram_quantileover them reads above 5 s. A unit test checks this. - S03-B's bucket table and its rationale paragraph state the new bounds.
docs/dev/observability.mdanddocs/dev/database-query-patterns.mdagree with them. - Every comment listed above is rewritten to describe the new ceiling.
reconcileStatementTimeoutkeeps its value, and its comment no longer gives the bucket ceiling as a reason. - The #944 bucket question is answered on #944. #944's other half, "a parked statement exports nothing", stays open there.
- The #908 (closed) series-per-name figure is updated, and the cardinality audit passes.
Size and fit
About 100-200 lines, mostly docs and comments, with one code line. This fits one smurfit unit.
Dependencies
None blocking. It edits the same S03-B bucket table and the same metrics.go block as #1719 (closed), so rebase whichever lands second. #1607's "queries past the 5 s ceiling" panel and #1664's deferred latency rule follow this issue.