Let the database query histogram tell a background statement past 5 s apart from one at 5 s

Split from #1606 (closed) (gap 3). Parent epic: &24088. Settles the bucket half of #944.

Decision (operator ruling 2026-10-25)

  • Q5, buckets: append 10, 30 to dbLatencyBuckets, so the family becomes 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30. This costs about 1,230 series per pod (2 bounds × about 616 names). It covers the longest production statement budget (TombstoneTimeout = 10 s) with headroom, in one family, with no catalog split. Transactions get their own wide family in #1719 (closed).
  • Q7, spec: this MR amends S03-B itself (the bucket family is spec-pinned at docs/specs/S03-b-metrics.md:461). #1718 (closed) lands first.

Problem

dbLatencyBuckets ends at 5 s (internal/metrics/metrics.go:41), and S03-B pins that family (docs/specs/S03-b-metrics.md:461). The following all land in +Inf, so histogram_quantile reads exactly 5 s for them: reap pages, reconciliation sums, drain lock waits, tombstone counts and the tombstone stamp under its 10 s budget (#944). The #1607 dashboard has to describe "a flat 5 s line means over 5 s".

Evidence

Research base: origin/main at e2c420456, 2026-10-25.

  • internal/metrics/metrics.go:41: dbLatencyBuckets = []float64{0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5}. All four query vectors use it (internal/metrics/database.go:42), and so does the remote negative-cache histogram through DBLatencyBuckets (:46).
  • Background statement budgets above or at 5 s: internal/lifecycle/tombstone.go:57 (TombstoneTimeout = 10 * time.Second), internal/accounting/reconcile_task.go:97 (reconcileStatementTimeout = 5 * time.Second), internal/accounting/backlog_collector.go:16, internal/accounting/reconcile_trigger.go:232.
  • Several comments tie a value or a claim to the 5 s ceiling. They must change in the same MR (guardrail 16):
    • internal/accounting/reconcile_task.go:91-95 ("Five seconds is also the largest bound the timing can still tell apart");
    • internal/accounting/reconcile_trigger.md:48;
    • internal/datastore/lifecycle_reap_maven.md:88, :96;
    • internal/datastore/reconcile_namespace.go:60-61;
    • internal/datastore/reconcile_repository.go:457.
  • Series cost: each name costs 15 series today, 12 of them _bucket (#908 (closed)). Each added bound costs about 616 series per pod.
  • The remote negative-cache histogram shares DBLatencyBuckets (internal/metrics/database.go:46), so it gains the two bounds too. Check that this is wanted, or give it its own copy of the old family.

Fix direction

  1. Append 10, 30 to dbLatencyBuckets in internal/metrics/metrics.go.
  2. Update the S03-B bucket table and its rationale paragraph (docs/specs/S03-b-metrics.md:458-471), docs/dev/observability.md and docs/dev/database-query-patterns.md.
  3. Rewrite every comment listed above to describe the new ceiling.

Acceptance criteria

  • A statement that takes 7 s and one that takes 25 s land in different buckets (le="10" and le="30"), and a histogram_quantile over them reads above 5 s. A unit test checks this.
  • S03-B's bucket table and its rationale paragraph state the new bounds. docs/dev/observability.md and docs/dev/database-query-patterns.md agree with them.
  • Every comment listed above is rewritten to describe the new ceiling. reconcileStatementTimeout keeps its value, and its comment no longer gives the bucket ceiling as a reason.
  • The #944 bucket question is answered on #944. #944's other half, "a parked statement exports nothing", stays open there.
  • The #908 (closed) series-per-name figure is updated, and the cardinality audit passes.

Size and fit

About 100-200 lines, mostly docs and comments, with one code line. This fits one smurfit unit.

Dependencies

None blocking. It edits the same S03-B bucket table and the same metrics.go block as #1719 (closed), so rebase whichever lands second. #1607's "queries past the 5 s ceiling" panel and #1664's deferred latency rule follow this issue.

Edited by Pawel Rozlach