## The Why <details><summary>Expand to read The Why</summary> GitLab's analytics APIs (pipelines, deployments, merge requests, Duo Agent Platform sessions, Duo workflows, AI usage events, code suggestions, contributions) let callers filter results by picking the values they want. There is no way to filter by exclusion. To exclude something today, a caller has to list every other value it does want. That only works when the list of possible values is small and fixed. It breaks down for things like branch names, project IDs, user IDs, or flow types, where the list is open-ended or grows over time as GitLab adds new statuses, events, or features. Common requests we can't satisfy today: - Exclude bot users from contribution counts. - Exclude `master`/`main` from a pipeline breakdown by branch. - Exclude a noisy flow type from Duo Agent Platform stats. - Exclude skipped pipelines from duration numbers. This issue adds the "exclude" capability to the underlying filtering system that powers these APIs. The follow-up issue https://gitlab.com/gitlab-org/gitlab/-/work_items/629296 will roll it out field by field. gitlab-org/gitlab#611201 rolled out a user filter and a dimension to every analytics field the same way, one framework change followed by a rollout issue. This issue follows the same pattern, but for exclusion filters. TBD: parent epic </details> ## Proposal Once this lands, engineers who own an analytics field can add an "exclude these values" filter with the same effort it takes to add today's "include these values" filter. End users won't see a change yet from this issue alone. The follow-up issue decides which fields actually get an exclude option and adds it to each one. --- ## Technical Proposal Add a new filter type, `exact_not_match`, next to the existing `exact_match` filter in the ClickHouse aggregation engine. It behaves like `exact_match` but excludes the given values instead of including them (`NOT IN` instead of `IN`). It supports the same options (`merge_column`, `formatter`, `max_size`) and needs its own identifier so it doesn't collide with the positive filter on the same column. Once registered, it shows up automatically as a GraphQL argument on every engine that declares it, no adapter changes needed. ## Acceptance criteria - [ ] A ClickHouse engine can declare an `exact_not_match` filter on a column (e.g. `status`), and a request excluding one or more values (e.g. `status_not: ['skipped']`) removes matching rows from every metric in the response. - [ ] The filter works both as a `WHERE` condition and, for columns that need `merge_column: true`, as a `HAVING` condition. - [ ] The filter works correctly when applied to a table's primary key column, same as the existing include filter. - [ ] `formatter` and `max_size` options work the same way they do for the existing include filter. - [ ] Any engine that declares the filter exposes it in GraphQL as a list argument named after the column with a `Not` suffix (e.g. `statusNot`), without any changes to the GraphQL layer. - [ ] Developer documentation describes the new filter, its naming convention, and how it treats missing (NULL) values. - [ ] New and updated tests for the filter, the engine's filter registry, and the GraphQL argument pass. ## Out of scope - Rolling this filter out to any existing analytics field. That's the follow-up issue. - A DSL helper that declares the include and exclude filter together in one line. Also the follow-up issue. - An exclusion filter for metric values (the `HAVING`-only counterpart to `metric_exact_match`). - Changing how the filter treats rows with a missing (NULL) value in the filtered column. For this first version, those rows are simply dropped by the exclude filter; see the NULL note below. - Adding this filter to the PostgreSQL-backed aggregation engine. No production analytics field uses that backend today. <details><summary>Implementation details</summary> - New class `Gitlab::Database::Aggregation::ClickHouse::ExactNotMatchFilter < FilterDefinition` in `lib/gitlab/database/aggregation/click_house/exact_not_match_filter.rb`. Mirrors `lib/gitlab/database/aggregation/click_house/exact_match_filter.rb`, but builds `column(query_builder).not_in(filter_config[:values])` instead of `.in(...)`. Still uses `having` when `merge_column?` is true, `where` otherwise. - Override `identifier` on the new class to return `:"#{name}_not"`. `name` still points at the real column (from `PartDefinition` in `lib/gitlab/database/aggregation/part_definition.rb`), so the filter targets the right column while getting a unique identifier. This is required because `Gitlab::Database::Aggregation::Engine::Dsl#guard_definitions_uniqueness!` rejects two filters with the same identifier, and a column usually already has a positive `exact_match` filter using the plain column name. - Register the new class in `filters_mapping` in `lib/gitlab/database/aggregation/click_house/engine.rb`, under the key `exact_not_match`. - No changes needed in `apply_inner_filters` / `build_base_query` / `pk_filter?` (same file): they key off `filter.definition.name`, which is unchanged, so a negative filter on a primary key column is placed in the dedup subquery automatically, same as the positive filter. - No changes needed in `lib/gitlab/database/aggregation/graphql/adapter.rb`: both `filter_to_arguments` and `build_filter` have generic `else` branches keyed off `filter.identifier` that already handle this class correctly. Add a spec case to `spec/lib/gitlab/database/aggregation/graphql/adapter_spec.rb` to lock this in. - No changes needed in `lib/gitlab/database/aggregation/query_plan.rb` or `lib/gitlab/database/aggregation/query_plan/filter.rb`: both are keyed off `identifier`, so `formatter:` and `max_size:` keep working. - ClickHouse gotcha: `NOT IN` on a nullable column also drops rows where the column is NULL, because `NULL NOT IN (...)` evaluates to NULL, which `WHERE`/`HAVING` treat as false. Keep this behavior for the first iteration (simpler, matches how `NOT IN` normally works), but document it clearly and flag it as a possible follow-up if teams need NULL rows included. - Docs: update `doc/development/aggregation_engines.md`. Add a `#### exact_not_match filter` section next to the existing `#### exact_match filter` section (around line 578), covering the `_not` identifier suffix and the NULL behavior. Update the "Filter placement" section (around line 1015) so it lists `exact_not_match` alongside `exact_match` and `range` as filters applied to the base query rather than after aggregation. - Tests: add `spec/lib/gitlab/database/aggregation/click_house/exact_not_match_filter_spec.rb`, mirroring `spec/lib/gitlab/database/aggregation/click_house/exact_match_filter_spec.rb` (same shared context, inline test engine). Cover: excluding a single value, excluding multiple values, the `merge_column: true` / `HAVING` case, the identifier resolving to `<name>_not`, and `max_size` validation still triggering. Add a mapping assertion to `spec/lib/gitlab/database/aggregation/click_house/engine_spec.rb`. </details>