Add exact_not_match filter to ClickHouse aggregation engines

The Why

Expand to read The Why

GitLab's analytics APIs (pipelines, deployments, merge requests, Duo Agent Platform sessions, Duo workflows, AI usage events, code suggestions, contributions) let callers filter results by picking the values they want. There is no way to filter by exclusion.

To exclude something today, a caller has to list every other value it does want. That only works when the list of possible values is small and fixed. It breaks down for things like branch names, project IDs, user IDs, or flow types, where the list is open-ended or grows over time as GitLab adds new statuses, events, or features.

Common requests we can't satisfy today:

  • Exclude bot users from contribution counts.
  • Exclude master/main from a pipeline breakdown by branch.
  • Exclude a noisy flow type from Duo Agent Platform stats.
  • Exclude skipped pipelines from duration numbers.

This issue adds the "exclude" capability to the underlying filtering system that powers these APIs. The follow-up issue https://gitlab.com/gitlab-org/gitlab/-/work_items/629296 will roll it out field by field.

gitlab-org/gitlab#611201 rolled out a user filter and a dimension to every analytics field the same way, one framework change followed by a rollout issue. This issue follows the same pattern, but for exclusion filters.

TBD: parent epic

Proposal

Once this lands, engineers who own an analytics field can add an "exclude these values" filter with the same effort it takes to add today's "include these values" filter. End users won't see a change yet from this issue alone. The follow-up issue decides which fields actually get an exclude option and adds it to each one.

Technical Proposal

Add a new filter type, exact_not_match, next to the existing exact_match filter in the ClickHouse aggregation engine. It behaves like exact_match but excludes the given values instead of including them (NOT IN instead of IN). It supports the same options (merge_column, formatter, max_size) and needs its own identifier so it doesn't collide with the positive filter on the same column. Once registered, it shows up automatically as a GraphQL argument on every engine that declares it, no adapter changes needed.

Acceptance criteria

  • A ClickHouse engine can declare an exact_not_match filter on a column (e.g. status), and a request excluding one or more values (e.g. status_not: ['skipped']) removes matching rows from every metric in the response.
  • The filter works both as a WHERE condition and, for columns that need merge_column: true, as a HAVING condition.
  • The filter works correctly when applied to a table's primary key column, same as the existing include filter.
  • formatter and max_size options work the same way they do for the existing include filter.
  • Any engine that declares the filter exposes it in GraphQL as a list argument named after the column with a Not suffix (e.g. statusNot), without any changes to the GraphQL layer.
  • Developer documentation describes the new filter, its naming convention, and how it treats missing (NULL) values.
  • New and updated tests for the filter, the engine's filter registry, and the GraphQL argument pass.

Out of scope

  • Rolling this filter out to any existing analytics field. That's the follow-up issue.
  • A DSL helper that declares the include and exclude filter together in one line. Also the follow-up issue.
  • An exclusion filter for metric values (the HAVING-only counterpart to metric_exact_match).
  • Changing how the filter treats rows with a missing (NULL) value in the filtered column. For this first version, those rows are simply dropped by the exclude filter; see the NULL note below.
  • Adding this filter to the PostgreSQL-backed aggregation engine. No production analytics field uses that backend today.
Implementation details
  • New class Gitlab::Database::Aggregation::ClickHouse::ExactNotMatchFilter < FilterDefinition in lib/gitlab/database/aggregation/click_house/exact_not_match_filter.rb. Mirrors lib/gitlab/database/aggregation/click_house/exact_match_filter.rb, but builds column(query_builder).not_in(filter_config[:values]) instead of .in(...). Still uses having when merge_column? is true, where otherwise.
  • Override identifier on the new class to return :"#{name}_not". name still points at the real column (from PartDefinition in lib/gitlab/database/aggregation/part_definition.rb), so the filter targets the right column while getting a unique identifier. This is required because Gitlab::Database::Aggregation::Engine::Dsl#guard_definitions_uniqueness! rejects two filters with the same identifier, and a column usually already has a positive exact_match filter using the plain column name.
  • Register the new class in filters_mapping in lib/gitlab/database/aggregation/click_house/engine.rb, under the key exact_not_match.
  • No changes needed in apply_inner_filters / build_base_query / pk_filter? (same file): they key off filter.definition.name, which is unchanged, so a negative filter on a primary key column is placed in the dedup subquery automatically, same as the positive filter.
  • No changes needed in lib/gitlab/database/aggregation/graphql/adapter.rb: both filter_to_arguments and build_filter have generic else branches keyed off filter.identifier that already handle this class correctly. Add a spec case to spec/lib/gitlab/database/aggregation/graphql/adapter_spec.rb to lock this in.
  • No changes needed in lib/gitlab/database/aggregation/query_plan.rb or lib/gitlab/database/aggregation/query_plan/filter.rb: both are keyed off identifier, so formatter: and max_size: keep working.
  • ClickHouse gotcha: NOT IN on a nullable column also drops rows where the column is NULL, because NULL NOT IN (...) evaluates to NULL, which WHERE/HAVING treat as false. Keep this behavior for the first iteration (simpler, matches how NOT IN normally works), but document it clearly and flag it as a possible follow-up if teams need NULL rows included.
  • Docs: update doc/development/aggregation_engines.md. Add a #### exact_not_match filter section next to the existing #### exact_match filter section (around line 578), covering the _not identifier suffix and the NULL behavior. Update the "Filter placement" section (around line 1015) so it lists exact_not_match alongside exact_match and range as filters applied to the base query rather than after aggregation.
  • Tests: add spec/lib/gitlab/database/aggregation/click_house/exact_not_match_filter_spec.rb, mirroring spec/lib/gitlab/database/aggregation/click_house/exact_match_filter_spec.rb (same shared context, inline test engine). Cover: excluding a single value, excluding multiple values, the merge_column: true / HAVING case, the identifier resolving to <name>_not, and max_size validation still triggering. Add a mapping assertion to spec/lib/gitlab/database/aggregation/click_house/engine_spec.rb.
Edited by 🤖 GitLab Bot 🤖