Add traversal_path dimension to aggregation engines

Brandon Labuschagnerequested to merge
605518-ai-usage-events-group-dimension-depth into master

What does this MR do and why?

This MR adds a traversal_path dimension to the ClickHouse aggregation-engine framework. It is framework-only: no engine adopts the dimension in this MR.

The framework now provides:

  • A new TraversalPathDimension class (lib/gitlab/database/aggregation/click_house/traversal_path_dimension.rb), usable via the traversal_path DSL keyword inside an engine's dimensions do ... end block.
  • Registration of that keyword in dimensions_mapping (lib/gitlab/database/aggregation/click_house/engine.rb).
  • TraversalPathDimension expects an organization-scoped traversal path column, such as 7/12/34/, where the first segment is the organization id. It skips that segment, so depth: 1 (the default) is the top-level group, depth: 2 the second-level subgroup, and so on. The SQL shape is nullIf(toUInt64OrNull(arrayElement(splitByChar('/', <path column>), depth + 1)), 0). Paths shorter than the requested depth, and the 0/ placeholder path, resolve to a NULL bucket.
  • depth is validated against 1..MAX_DEPTH (MAX_DEPTH = 99) with the framework's existing in: mechanism. An engine can narrow the range with parameters: { depth: { in: 1..2 } }.
  • Support for parameterized association dimensions in GraphQL: a dimension's parameters (like depth) become GraphQL field arguments, and the row is read via the parameterized instance key, so a query like group(depth: 2) { id } resolves correctly. declare_association_field moved into BaseResponseType, next to declare_parameterized_field, and both now share a private declare_parameter_arguments helper.
  • The association: Hash gains a preloader key alongside the existing model, graphql_type, and finder keys. The framework calls preloader.new(records, current_user).execute once per batch after loading, so an engine can avoid N+1 queries on association fields. current_user travels through the batch loader key to support this.
  • Documentation for the dimension and for the finder/preloader options in doc/development/aggregation_engines.md, including a note that orderBy must pass the same parameters as the field selection.

Why

The "Duo adoption by group" dashboard chart needs distinct AI users per group (usersCount grouped by group), and per-group trends when combined with the existing timestamp date bucket. A first attempt, !250645 (closed), grouped by the last segment of the traversal path and was closed after review, because that segment is the project namespace id for events tracked in a project, not a group id. The reviewer proposed a depth parameter instead, which is what this dimension implements.

The same design will be reused for the DuoWorkflows engine, the next planned consumer of this framework class, written up in https://gitlab.com/gitlab-org/gitlab/-/work_items/617805.

Notes for reviewers

Three things changed since the previous revision of this MR, in response to review:

  1. Removed the with_organization: option entirely. The dimension now always skips the leading organization segment. The engine reads an org-scoped traversal_path column, so no engine needs with_organization: false, and the next planned consumer (DuoWorkflows) already stores org-prefixed paths too. Dropped the docs row and the spec case for the option; the spec fixture now uses org-scoped paths throughout.
  2. Removed the validate_part override that checked "depth must be a positive integer", along with its translatable string in locale/gitlab.pot. The existing in: 1..MAX_DEPTH range already rejects 0, negative values, and values above the maximum. The spec now covers 0, -1, and 100, all rejected with "Invalid value(s) for parameter depth". The only value the range let through was a non-integer Float, because Range#include? uses cover semantics for numeric ranges ((1..99).include?(1.5) is true). That input is unreachable in practice, since depth is a GraphQL Int argument and the only code that builds an aggregation Request is the two GraphQL resolvers.
  3. Replaced Namespace::NUMBER_OF_ANCESTORS_ALLOWED + 1 with a framework constant, MAX_DEPTH = 99, on TraversalPathDimension, so lib/gitlab/database/aggregation has no reference to application models or constants. Specs cover depth 99 accepted and depth 100 rejected. The docs row now says 1 to 99.

This MR is now framework-only, force-pushed as a single commit, a390256a, 7 files, +326/-40. The Analytics::AggregationEngines::AiUsageEvents adoption moved to a separate draft MR, !254731 (merged), stacked on this branch (it will be retargeted to master after this MR merges). Merge order: this MR first, then !253365 (merged) (rebuilds ai_usage_events with traversal_path as the leading sort key), then MR 254731.

No engine adopts the dimension in this MR, so there is no GraphQL schema change and no translatable strings added. The Changelog trailer is omitted because nothing user-facing changes until an engine adopts the dimension.

References

Screenshots or screen recordings

Backend only, no screenshots.

How to set up and validate locally

Run the two spec files:

bundle exec rspec spec/lib/gitlab/database/aggregation/click_house/traversal_path_dimension_spec.rb
bundle exec rspec spec/graphql/types/analytics/aggregation/engine_response_type_spec.rb

19 examples pass locally. There is no GraphQL field to query end-to-end until an engine adopts the dimension; see !254731 (merged) for that.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Brandon Labuschagne

Merge request reports