Raise the ClickHouse aggregation page size cap to 500

Brandon Labuschagnerequested to merge
630507-clickhouse-aggregation-page-size-500 into master

What does this MR do and why?

This MR raises Gitlab::Database::Aggregation::ClickHouse::Engine.max_page_size from 250 to 500. That is the whole code change.

The cap only applies while the larger_clickhouse_aggregation_pages feature flag is enabled for the current user. The flag was enabled on gitlab.com on 2026-09-30 at 08:12 UTC for the user actor. Without the flag, AggregationConnection.effective_max_page_size returns the schema default of 100.

Bigger pages mean fewer requests per panel. Each page re-runs the whole ClickHouse aggregation and skips earlier rows with OFFSET. AggregationConnection#load_nodes asks for limit plus one rows to detect a next page. GLQL selects count on every page, and since !257204 (merged) the count is derived from the last page, but any page that has a next page still runs a separate COUNT aggregation over the same GROUP BY. So a panel with N rows and page size P costs ceil(N/P) GraphQL requests and 2 * ceil(N/P) - 1 ClickHouse aggregations.

Other changes in this MR:

  • The feature flag YAML description for larger_clickhouse_aggregation_pages now says 500.
  • aggregation_field_resolver_spec.rb asserts the cap is 500.
  • The complexity table in aggregation_scope_type_spec.rb expects 60 for an omitted first and for first: 1000 with the flag on. It keeps 35 for first: 250.

GLQL reads the cap from gon.aggregation_max_page_size, which Gitlab::GonHelper sets from the engine while the flag is on. In app/assets/javascripts/glql/components/common/resolver.vue, auto-paginated aggregated displays request pages of limit: <cap>. These are the chart display types on analytics dashboards. They walk pages until hasNextPage is false or 1000 rows, the MAX_AUTO_PAGINATED_ROWS constant. At 500 the page loop needs at most 2 requests instead of 4.

The previous-period comparison behind showTrends runs once as a single page at the cap. The resolver discards its result when the main result's count is larger than the cap. With a 500 cap, panels with 251 to 500 rows keep their trend comparison. At 250 they lose it.

Why 500 is safe

  • GraphQL complexity. Complexity for this field is 10 * (1 + rows / 100): 60 at 500 rows, 35 at 250, 20 at 100. The authenticated limit is 250 and the default is 200. GLQL sends one aggregated connection per request, and the comparison is its own request.
  • Rollback. With the flag off, the effective page cap and the complexity both fall back to the schema default of 100 through AggregationConnection.effective_max_page_size. Types::Analytics::Aggregation::AggregationField uses the same flag-aware cap. Disabling the flag is the rollback. No data is written.
  • ClickHouse. Per-query work is unchanged. LIMIT only truncates the output of the GROUP BY, so ClickHouse computes the same aggregation state whatever the page size. Timeouts are unchanged at 20 s for the HTTP read timeout to ClickHouse and 30 s for the GraphQL timeout.
  • Rails and response size. On the Rails side each extra row costs up to about 3 ms for date and string dimensions, and about 100 to 180 bytes of JSON. A 500-row page adds at most about 0.7 s and 25 to 45 KB to one request compared with a 250-row page, and it replaces a second request plus a COUNT aggregation.
  • Other clients that omit first. The AI impact dashboard fragments for code suggestions by language, by IDE, and for monthly pipelines return a handful of rows. The Duo feature retention query pages with after and needs fewer pages. External GraphQL API callers with the flag enabled get up to 500 rows per page when they omit first.
  • Frontend page loop. MAX_AUTO_PAGINATED_ROWS stays 1000, so the request cap per panel drops from 4 to 2.
  • Specs. The two changed spec files plus aggregation_connection_spec.rb and gon_helper_spec.rb pass locally on 76fc9c17, with 81 examples and 0 failures. The remaining literal 250s in specs build a connection with an explicit max_page_size: 250 or set gon directly. They do not depend on the engine constant.

Expected effect

Total ClickHouse work falls for any panel with more than 250 rows. Per-query CPU and memory do not change.

Row counts on the DAP Impact dashboard for gitlab-org, last 30 days. This is the default range with daily buckets, 31 days and 29 flow types.

Panel Tab Rows Requests at 100 / 250 / 500 ClickHouse aggregations at 100 / 250 / 500
Sessions over time, created daily by flowType Work 460 5 / 2 / 1 9 / 3 / 1
Credits over time, same dimensions with creditsUsedSum Spend 460 groups, same GROUP BY as Sessions over time 5 / 2 / 1 9 / 3 / 1
Sessions by tier and top capabilities, userTier by flowType heat map Adoption 91 1 / 1 / 1 1 / 1 / 1
Credits by model, by capability, model by flowType Spend 29 1 / 1 / 1 1 / 1 / 1
Bar lists by flowType Overview, Adoption, Work 29 1 1
Every date-only line chart Overview, Work 31 or fewer 1 1
Share of users vs share of sessions, New vs returning Adoption 4 and 2 1 1

The Credits over time metric could not be run through the API by the author. credits_used.sum requires read_agent_artifacts. Its row count is inferred from having the same dimensions as Sessions over time. Every other row count was measured.

Baseline is the 2026-09-30 browser rerun on gitlab.com with the 250 cap live, linked in References.

Tab Requests at 250 Requests at 500 Change
Overview 12 12 none
Adoption 15 15 none, the comparison request grows to 500 rows
Work, first view 17 16 Sessions over time 2 to 1
Spend 7 7 none for the rerun user, see below

Work also sent 16 requests when scrolled to the bottom. Those are date-only line charts, so that batch is unchanged.

The rerun user cannot read creditsUsedSum on gitlab-org, which needs read_agent_artifacts, so the Spend credit panels returned an error in one request each and never paged. For a user who can read credits, Credits over time has the same 460 groups and goes from 2 requests to 1 as well.

Per dashboard visit at this scope, this is 1 fewer GraphQL request and 2 fewer ClickHouse aggregations for the rerun user, and 2 and 4 for a user who can read credits. Each affected panel should finish about 0.5 s sooner. Median request time on Work and Adoption should stay at 0.5 to 0.6 s. The one 500-row page is a slower single request, but it replaces two sequential ones.

For comparison, between the 2026-09-28 and 2026-09-30 reruns Work's first view went from 21 to 17 requests and its scroll-in batch from 19 to 16, and the median request time on Work and Adoption fell from 0.8 to 1.0 s to 0.5 to 0.6 s. That window contained the step from 100 to 250 in !257575 (merged) and !257576 (merged), the COUNT removal in !257204 (merged), and dashboard YAML changes, so the numbers cannot be attributed to the cap alone.

The gain is larger for scopes with panels in the 251 to 500 row band. Examples are more flow types over 31 daily buckets, or 91-day ranges with 14 weekly buckets. There, 3 aggregations become 1, 2 requests become 1, and the trend comparison is kept instead of discarded. Panels above 500 rows still page, up to the unchanged 1000-row ceiling, in 2 requests instead of 4.

One request gets heavier with no benefit. The Group and project comparison table on Adoption is a load-more display. Its main query asks for 20 rows and is not affected. Its previous-period comparison returns a full page at the cap, so it goes from 250 to 500 rows. It is discarded at both caps because the panel has 646 rows. For gitlab-org that is about 45 KB growing to about 90 KB. Wall time grows from about 1.6 s to an estimated 1.8 to 2.0 s, extrapolated from the flat measurements below. In the 2026-09-30 browser rerun it was already the slowest request on Adoption at 2.4 s. A follow-up could skip or cap the comparison for load-more displays. That is out of scope here.

Server load

What grows per request is Rails-side row handling and response bytes. ClickHouse work per query does not grow. The measurements below were taken through the GraphQL API, not in the browser, and every 500-row figure is an extrapolation because the cap on gitlab.com is still 250.

Measurement method

Measured on gitlab.com on 2026-09-30 as blabuschagne with the cap at 250. Each row was run twice with glab api graphql. Wall clock includes the network round trip from a laptop.

The query shape was:

group(fullPath: "gitlab-org") {
  analytics {
    duoWorkflows(createdAtFrom: "2026-08-31T00:00:00Z", createdAtTo: "2026-09-30T23:59:59Z") {
      aggregated(first: N) {
        count
        pageInfo { hasNextPage endCursor }
        nodes { ... }
      }
    }
  }
}
Sessions over time query

Dimensions are createdAt(granularity: "daily") and workflowDefinition. The metric is totalCount.

first wall time, 2 runs response bytes
100 0.95 s, 1.10 s 10,034
250 1.31 s, 1.54 s 24,981

A stat query with no dimensions and one row took 0.65 s and 0.68 s. So the per-row cost on the Rails side is roughly 2.5 to 3 ms and about 100 bytes of JSON.

A 500-row page is extrapolated to about 2.0 s and about 46 KB in a single request. These two numbers are not measured.

Walking the same 460 rows in two 250-row pages today measured 1.26 s then 1.21 s. The pages run one after the other because the second needs the first page's cursor. That is about 2.5 s for the panel. The expected saving is about 0.5 s of wall time, one request, and two ClickHouse aggregations, the second data page and the COUNT.

Group and project comparison query

Dimensions are group(depth: 2) { fullPath } and project { fullPath }. Metrics are usersCount and creditsUsed { sum }. The result has 646 rows.

first wall time, 2 runs response bytes
20 1.47 s, 1.50 s 3,836
100 1.44 s, 1.68 s 18,260
250 1.58 s, 1.72 s 45,235

Wall time is flat in rows because the group and project associations are batch loaded. Size is about 178 bytes per row, so a 500-row page is about 90 KB. That figure is extrapolated.

Rollout issue 630765 still says 250 in its title and in "What could go wrong". It needs an edit when this merges. Monitoring stays as listed there. That covers ClickHouse query duration and memory, and GraphQL request duration and errors for the aggregation queries.

References

Screenshots or screen recordings

Not applicable, no visual change.

How to set up and validate locally

  1. In bin/rails console, run Feature.enable(:larger_clickhouse_aggregation_pages).
  2. Open an analytics dashboard with GLQL panels backed by ClickHouse. One example is the DAP Impact dashboard at /explore/analytics_dashboards/dap_impact?scope=<top-level group>. In the DevTools console, check that gon.aggregation_max_page_size is 500. On master it is 250.
  3. In DevTools, open the Network tab and filter on /api/glql. Auto-paginated aggregated queries send limit: 500 in their variables.
  4. With the flag disabled, the value is absent from gon and pages stay at the schema default of 100.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Brandon Labuschagne

Merge request reports