Add tracked-ref filtering and collapse to ES dependencies finder

Rushik Subbarequested to merge
627902-es-dependencies-finder-tracked-refs into master

What does this MR do and why?

Search::AdvancedFinders::Sbom::DependenciesFinder skipped the trackedRefIds and trackedRefsScope filters. The sbom_occurrence_refs index stores one document per (occurrence, tracked ref), so selecting several refs matches several documents for the same occurrence. De-duplication in Search::Elastic::ResponseMapper is per response, not across pages, which shortens pages, makes hasNextPage under-report, and repeats an occurrence that straddles a page boundary. Collapsing on sbom_occurrence_id returns one hit per occurrence instead.

Requesting either filter now routes to Elasticsearch on its own rather than only alongside malware. That is gated on the existing project_dependency_tracked_ref flag, which already decides whether the UI offers the filter token, so the frontend cannot offer a filter the backend would reject.

Note

Before this MR, a caller passing trackedRefIds to the project dependency list got a silently unfiltered list, because ::Sbom::DependenciesFinder ignores the argument. After it, with project_dependency_tracked_ref off or Elasticsearch unavailable, they get The tracked_ref_ids filter is not available. Disabling the flag therefore does not restore the previous behaviour — it swaps a silent wrong answer for a loud refusal. The UI is unaffected either way, since it gates the filter token on the same flag.

Collapse Verification

Verification

Collapse changes no result set, fixes short pages on every filter, and regresses nothing else.

How this was tested

use_elasticsearch? was temporarily forced to route every project dependency request to Elasticsearch, so filters that normally stay on PostgreSQL were exercised against the sbom_occurrence_refs index. The change was reverted afterwards and is not part of this MR.

gitlab-org/malicious-packages-test-repo (29 occurrences) was run twice:

  • single-ref — its one default tracked ref, 29 documents
  • multi-ref — 87 documents across 3 contexts (2 default, 1 non-default), so every occurrence has three ref documents and two of them match the default-branch filter

::Sbom::DependenciesFinder (PostgreSQL) was the oracle in both runs.

Collapse never changes the result set

All 16 filter and sort combinations returned identical sets across PostgreSQL, Elasticsearch with collapse, and Elasticsearch without collapse, in both the single-ref and multi-ref runs. No duplicates in any of them.

Combinations covered

no filter, component_names (one and three values), component_ids, component_versions, not_component_versions, package_managers, source_types (defaults, nil_source, and an unknown value), licenses, and sorting by name ascending and descending, severity, packager and license.

The short-page bug affects every filter, not only the tracked-ref ones

Asking for 5 rows against the multi-ref data:

Filter With collapse Without
no filter 5 3
component_names (10 values) 5 3
not_component_versions 5 3
licenses 5 3
source_types 5 3
sort by severity, descending 5 3
sort by packager, ascending 5 3
sort by license, descending 5 3

Always 3, because 5 documents spread over 2 default refs cover only 3 occurrences (2 + 2 + 1) before ResponseMapper de-duplicates them.

Two things follow. The bug only shows on a bounded page, which is what every real request is: an earlier run using first(200) showed no difference at all, because a page larger than the whole result set lets per-response de-duplication hide it. And the fix is no, since any project with two default contexts was already getting short pages on ordinary filters.

Nothing else regressed

  • Keyset pagination. A forward walk at 4 rows per page, across all 8 filter and sort combinations, covered the full set exactly, with no gaps and no repe
  • Aggregations. The aggregation query comes back with size: 0, no collapse key and its aggs intact, so skip_if_size_zero keeps collapse off that path a
  • GraphQL, end to end, with routing forced to Elasticsearch: componentNames returned 1 node, first: 5 returned 5, and sort: NAME_ASC returned 20. No errors a

References

Resolves #627902 (closed)

Screenshots or screen recordings

Before After

How to set up and validate locally

Prerequisites

  1. Ultimate licence, so dependency_scanning and security_dashboard are enabled.

  2. Elasticsearch running, with both Elasticsearch indexing and Search with Elasticsearch enabled turned on under Admin > Settings > Advanced Search.

  3. Confirm both index migrations have finished, in the rails console:

    Elastic::DataMigrationService.migration_has_finished?(:create_sbom_occurrence_refs_index)
Elastic::DataMigrationService.migration_has_finished?(:backfill_sbom_occurrence_refs_index)

    Both must return true, otherwise the dependency list stays on PostgreSQL.

  4. Enable the flag. It is off by default and gates everything below:

    Feature.enable(:project_dependency_tracked_ref)

    vulnerabilities_across_contexts is already default-on; only re-enable it if you turned it off locally.

Data setup

  1. Run a dependency-scanning pipeline on the project's default branch and on at least one other branch. That is what creates the Security::ProjectTrackedContext and Sbom::OccurrenceRef rows. A project is capped at 4 tracked refs.

  2. Give the project a second default context. This is the case the collapse fixes, and nothing in the schema prevents it: the unique index covers (project_id, context_name, context_type), not is_default. It has to be a branch, because only_branch_refs_can_be_default bars tags:

    project = Project.find_by_full_path('group/project')
Security::ProjectTrackedContext.for_project(project.id)
  .non_default_refs.for_context_type(:branch).first.update!(is_default: true)

  3. Re-index after any console change to contexts or refs:

    Sbom::BulkEsOperationService.new(Sbom::OccurrenceRef.by_project(project.id)).execute
Elastic::ProcessBookkeepingService.new.execute

  4. List the ref global IDs you will filter on:

    Security::ProjectTrackedContext.for_project(project.id)
  .map { |c| [c.context_name, c.is_default, c.to_global_id.to_s] }

Checks

Run these at /-/graphql-explorer.

  1. Several refs, one row each. Filter on two refs that both contain the same dependency. Every dependency appears exactly once, and the page fills to first: rather than coming back short.

    query {
  project(fullPath: "group/project") {
    dependencies(
      trackedRefIds: [
        "gid://gitlab/Security::ProjectTrackedContext/1"
        "gid://gitlab/Security::ProjectTrackedContext/2"
      ]
      first: 20
    ) {
      pageInfo { hasNextPage endCursor }
      nodes { id name version }
    }
  }
}

  2. All refs. Same query with trackedRefsScope: ALL_REFS and no trackedRefIds. Same expectation: one row per dependency.

  3. Two default contexts, no filter. Drop both tracked-ref arguments. Each dependency still appears once. This is what regressed before the collapse.

  4. Pagination. Walk the list with first: 2, following endCursor into after:. No dependency is skipped, none appears on two consecutive pages, and hasNextPage goes false only on the last page.

  5. Flag off. Feature.disable(:project_dependency_tracked_ref), then repeat check 1. Returns The tracked_ref_ids filter is not available. These filters are answerable only by Elasticsearch, so the request is rejected rather than served an unfiltered list. Re-enable afterwards.

  6. Index unavailable. Turn off Search with Elasticsearch enabled. Check 1 now returns The tracked_ref_ids filter is not available. instead of silently returning an unfiltered list.

  7. Conflicting filters. policyViolations: [DISMISSED_IN_MR] with trackedRefIds returns The policy_violations filter cannot be combined with the tracked_ref_ids filter.

  8. Malware guard unchanged. malware: true with trackedRefsScope: ALL_REFS still returns The malware filter cannot be combined with non-default tracked refs.

UI

With the flag on, Project > Secure > Dependency list shows the tracked-ref token in the filtered search bar. Selecting more than one ref must not produce duplicate rows.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #627902 (closed)

Edited by Rushik Subba

Merge request reports