Add exact_not_match filter to Aggregation Engines

What does this MR do and why?

Aggregation Engines power GitLab's ClickHouse-backed analytics GraphQL APIs (pipelines, deployments, merge requests, Duo Agent Platform sessions, AI usage events, and more). Today their filters can only include values, so there is no way to ask for "everything except X". That blocks requests like excluding bot users from contribution counts, excluding master/main from pipeline breakdowns by branch, or excluding skipped pipelines from duration metrics.

This MR adds a new exact_not_match filter type that behaves like the existing include filter but generates NOT IN instead of IN. It supports the same options (merge_column, formatter, max_size, custom expression) and works on primary key columns. It also adds an exact_match_pair DSL helper to declare both the include and exclude filter on a column in one line. No GraphQL adapter changes were needed: the new filter surfaces automatically as a camelCase list argument with a Not suffix (for example statusNot).

This is a framework change only; no analytics field is switched over yet. Rolling the filter out to specific fields is tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/629296.

Changelog: added

References

Screenshots or screen recordings

Backend-only change, no UI impact.

How to set up and validate locally

This requires ClickHouse running in GDK.

  1. Run the new and updated specs:

    bundle exec rspec spec/lib/gitlab/database/aggregation/click_house/exact_not_match_filter_spec.rb \
      spec/lib/gitlab/database/aggregation/definitions_collector_spec.rb \
      spec/lib/gitlab/database/aggregation/click_house/engine_spec.rb \
      spec/lib/gitlab/database/aggregation/graphql/adapter_spec.rb
  2. Optionally, confirm the generated SQL in a Rails console:

    engine_class = Gitlab::Database::Aggregation::ClickHouse::Engine.build do
      self.table_name = 'agent_platform_sessions'
    
      metrics { count }
    
      filters do
        exact_match_pair :flow_type, :string
      end
    end
    
    scope = ClickHouse::Client::QueryBuilder.new('agent_platform_sessions')
    engine = engine_class.new(context: { scope: scope })
    
    request = Gitlab::Database::Aggregation::Request.new(
      filters: [{ identifier: :flow_type_not, values: ['chat'] }],
      metrics: [{ identifier: :total_count }]
    )
    
    result = engine.execute(request)
    result[:data].send(:query).to_sql  # contains `flow_type` NOT IN ('chat')
    result[:data].to_a                 # runs the query against ClickHouse

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #629295 (closed)

Edited by Pavel Shutsin

Merge request reports

Loading
Loading