Add exact_not_match filter to Aggregation Engines
What does this MR do and why?
Aggregation Engines power GitLab's ClickHouse-backed analytics GraphQL APIs (pipelines, deployments, merge requests, Duo Agent Platform sessions, AI usage events, and more). Today their filters can only include values, so there is no way to ask for "everything except X". That blocks requests like excluding bot users from contribution counts, excluding master/main from pipeline breakdowns by branch, or excluding skipped pipelines from duration metrics.
This MR adds a new exact_not_match filter type that behaves like the existing include filter but generates NOT IN instead of IN. It supports the same options (merge_column, formatter, max_size, custom expression) and works on primary key columns. It also adds an exact_match_pair DSL helper to declare both the include and exclude filter on a column in one line. No GraphQL adapter changes were needed: the new filter surfaces automatically as a camelCase list argument with a Not suffix (for example statusNot).
This is a framework change only; no analytics field is switched over yet. Rolling the filter out to specific fields is tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/629296.
Changelog: added
References
- Issue: #629295 (closed)
- Follow-up rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/629296
- Developer docs:
doc/development/aggregation_engines.md(updated in this MR)
Screenshots or screen recordings
Backend-only change, no UI impact.
How to set up and validate locally
This requires ClickHouse running in GDK.
-
Run the new and updated specs:
bundle exec rspec spec/lib/gitlab/database/aggregation/click_house/exact_not_match_filter_spec.rb \ spec/lib/gitlab/database/aggregation/definitions_collector_spec.rb \ spec/lib/gitlab/database/aggregation/click_house/engine_spec.rb \ spec/lib/gitlab/database/aggregation/graphql/adapter_spec.rb -
Optionally, confirm the generated SQL in a Rails console:
engine_class = Gitlab::Database::Aggregation::ClickHouse::Engine.build do self.table_name = 'agent_platform_sessions' metrics { count } filters do exact_match_pair :flow_type, :string end end scope = ClickHouse::Client::QueryBuilder.new('agent_platform_sessions') engine = engine_class.new(context: { scope: scope }) request = Gitlab::Database::Aggregation::Request.new( filters: [{ identifier: :flow_type_not, values: ['chat'] }], metrics: [{ identifier: :total_count }] ) result = engine.execute(request) result[:data].send(:query).to_sql # contains `flow_type` NOT IN ('chat') result[:data].to_a # runs the query against ClickHouse
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #629295 (closed)