Use quantileExactInclusive in the shared ClickHouse quantile metric

What does this MR do?

Changes the shared ClickHouse quantile metric of the aggregation framework to generate quantileExactInclusive(level)(column) instead of quantile(level)(column), and adds a spec that asserts the generated SQL names the new function.

Closes #632228

Why?

ClickHouse quantile uses reservoir sampling with a reservoir of 8192 values. Above that many values it is approximate and not deterministic, so the same percentile query can return different numbers on each run. We saw up to about 6% difference during UAT of the DAP Impact dashboard.

Why quantileExactInclusive?

  • It returns the exact value and interpolates linearly between neighbours. quantile applies the same interpolation when all values fit in its reservoir, so results for small datasets are unchanged. The existing spec that expects 84.0 for the 0.1 quantile of five values still passes.
  • quantileExact without the suffix returns the nearest element with no interpolation. It would change small-dataset results.
  • quantileDeterministic is stable but still sampled. It also needs a determinator column declared by every engine that uses the shared class, which is a wider API change.

Cost

Exact quantiles keep every value of the group in memory, O(n) per aggregation state, instead of a fixed 8192-sample reservoir. Each requested level is a separate state, since ClickHouse only fuses quantile* calls under optimize_syntax_fuse_functions, which is off by default.

The class is used by the pipelines (duration), deployments (duration), merge requests (time to merge) and agent platform sessions (duration) engines, and by every measurement, which expands to a <name>.quantile metric.

Every query is scoped to a namespace, but the date range filters are optional (required: false in lib/gitlab/database/aggregation/graphql/adapter.rb), so the worst case is a large namespace with no date filter: tens of millions of 8-byte values, hundreds of megabytes for one state. The reason this is still acceptable is relative: the inner GROUP BY ALL subquery in lib/gitlab/database/aggregation/click_house/engine.rb already materializes one row per key over the same row set, with far more than 8 bytes per row, so exact quantiles add a constant factor and do not change the query's memory class. lib/click_house/http_client.rb sets no per-query max_memory_usage, so the server profile default is the only cap. Reviewers, please confirm this trade-off.

quantileExactInclusive has existed since ClickHouse 20.x, so the 25.1 floor in .gitlab/ci/global.gitlab-ci.yml is not a concern.

How to test

  1. Pick a namespace with more than 8192 pipelines in the date window.
  2. Run the same percentile query twice through GraphQL, for example the pipelines aggregation with duration.quantile(quantile: 0.95).
  3. Confirm both runs return identical results.

Verification

  • Locally against ClickHouse 26.8: the quantile spec plus the pipelines, deployments, merge requests and agent platform sessions engine specs, 182 examples, 0 failures.
  • The new spec example fails on the old code, since quantile(0.5)( does not contain quantileExactInclusive(0.5)(.
  • GraphQL request specs cannot run on the author's machine and are left to CI. Every exact-equality quantile expectation in them is at level 0.5 on fixtures far below 8192 rows, where both functions interpolate identically.

References

  • ClickHouse docs: quantileExactInclusive and quantile (reservoir sampling, 8192 elements).
  • Engine reference entry updated in doc/development/aggregation_engines.md.
Edited by Chandra Saripaka

Merge request reports

Loading
Loading