Add descendants filter to aggregation engines

Brandon Labuschagnerequested to merge
605518-traversal-path-filter into master

What does this MR do and why?

Adds a descendants filter to the ClickHouse aggregation engine framework. Request validation resolves Group Global IDs into traversal paths. The filter lets ClickHouse aggregation engines filter events by one group and all of its descendants.

This MR was split out of !254120 (merged) at a reviewer's request. The companion MR !254110 (merged) got the same request.

  • lib/gitlab/database/aggregation/click_house/descendants_filter.rb (new): DSL keyword descendants inside filters do ... end. Renders OR-ed startsWith(<path column>, '<traversal path>') conditions, as a WHERE clause or a HAVING clause with merge_column: true, like the sibling exact_match and range filters. Request validation resolves Group Global IDs to traversal paths with one PostgreSQL query, Group.id_in, and traversal_path(with_organization:). max_size is validated first, so an oversized request never reaches PostgreSQL. Validation rejects the whole request with "Values must be Global IDs of existing groups for filter <key>" when the value list is empty, or when a value is blank, is not a Global ID, belongs to another model such as a Project, or does not resolve to an existing group. Validation also rejects the request with "Values must be traversal paths ending with / for filter <key>" when a resolved path does not end with /, so a broken ID-to-path transformation cannot widen a match to an unrelated group. There is no 1=0 fallback, because validation guarantees at least one valid path.
  • lib/gitlab/database/aggregation/click_house/engine.rb: registers descendants: DescendantsFilter in filters_mapping.
  • The descendants :group_id filter for the CodeSuggestions and AgentPlatformSessions engines, and the regenerated GraphQL reference doc and introspection result, moved to a separate, stacked MR on branch 605518-duo-engines-descendants-filter: !256098 (closed).

Developer docs are updated in doc/development/aggregation_engines.md with a new descendants filter section.

Why

Drilling into one group on the Duo adoption dashboard needs its events plus events from all descendants. That includes subgroups and projects. An exact match on a namespace id cannot express that. Prefix-matching the path column with the group's traversal path can. It uses the table's primary key, the same way the engine's base scope does.

Notes for reviewers

  • Validation runs one PostgreSQL primary key lookup (Group.id_in) per request, before ClickHouse is queried. It does not check the current user's access to the groups. It does not need to, because engines always apply their own authorized base scope too.
  • The SQL shape produced is one startsWith(path_column, '<traversal path>') condition per resolved path, joined with OR. Traversal paths end in /, so a group with id 1 cannot match 12/. Validation rejects any resolved path that does not end with /, so a broken transformation cannot widen the match to an unrelated group. Invalid Global IDs, such as ones belonging to another model or with no matching group, fail validation instead of being dropped.

Dependencies and merge order

This MR targets master and has no dependencies. !254120 (merged) is stacked on it and targets this branch until it merges. !254110 (merged) merged and added the traversal_path dimension. !256098 (closed) is also stacked on this branch, for the CodeSuggestions and AgentPlatformSessions engine additions, and will retarget to master once this MR merges.

References

Screenshots or screen recordings

Backend framework only, so there are no screenshots.

How to set up and validate locally

Run the spec for the filter validation:

bin/rspec spec/lib/gitlab/database/aggregation/click_house/descendants_filter_spec.rb

For an end-to-end GraphQL check, follow the validation steps in !254120 (merged), which exercises this filter through the AiUsageEvents engine.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Brandon Labuschagne

Merge request reports