Track sbom_occurrence_refs for ES indexing

Rushik Subbarequested to merge
srushik/602913_track-sbom-records-for-ES-after-ingestion into master

What does this MR do and why?

Start tracking sbom_occurrence_refs for ES indexing.

Wrap the Elasticsearch tracking added for Sbom::OccurrenceRef inserts/updates/deletes behind a new sbom_occurrence_ref_es_indexing feature flag, scoped to group level:

  • IngestReportSliceService#update_elasticsearch now checks the flag before tracking inserted/updated refs via BulkEsOperationService.
  • DeleteNotPresentOccurrencesService#track_elasticsearch_deletes now checks the same flag before queuing ES deletes via ProcessBookkeepingService.track!.

Updates specs accordingly:

  • Adds coverage for the new flag-gated tracking behavior in both services (enabled/disabled cases).
  • Tightens the 'does not sync with ES when no vulnerabilities' shared example to only assert no vulnerability records were tracked, since Sbom::OccurrenceRef deletions can legitimately call track! even when no vulnerabilities are involved.

References

How to set up and validate locally

  1. Check if sbom_occurrence_refs ES index exists, if not create it by running -

    Elastic::DataMigrationService[20260618151203].migrate

  2. Select a project with dependencies and enable the FF for it via Rails console

    Feature.enable(:sbom_occurrence_ref_es_indexing, project.group)

  3. Trigger SBOM ingestion by adding the Dependency Scanning CI template to a test project containing a manifest file (e.g. Gemfile.lock):

    include:
  - template: Security/Dependency-Scanning.gitlab-ci.yml

  4. After the pipeline finishes, check that refs landed in Postgres via gdk rails console:

    project = Project.find_by_full_path('your-group/your-project')
Sbom::OccurrenceRef.where(project_id: project.id).count

  5. Force the ES bookkeeping queue to drain instead of waiting for cron:

    Search::Elastic::IndexBulkCronWorker.new.perform

  6. Verify the document exists in Elasticsearch with the denormalized fields (vulnerability_count, highest_severity, etc.):

    curl -s "http://localhost:9200/_cat/indices?v" | grep -i sbom
curl -s "http://localhost:9200/sbom_occurrence_refs/_doc/<ref_id>" | jq

  7. To test the sibling-ref reindex path, re-run the pipeline after changing something that affects Sbom::Occurrence for a component already tracked on another branch/tag (for example hardcode reachability to 1 in ee/app/services/sbom/ingestion/tasks/ingest_occurrences.rb). Re-check that branch's ref document in ES for the updated value (no need to re-run that branch's own pipeline).

  8. To test deletes, re-run the pipeline after removing a dependency from the package file so DeleteNotPresentOccurrencesService runs. Confirm the ref row is gone from Postgres, then after the cron worker drains again, confirm the ES document for that id returns "found": false.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rushik Subba

Merge request reports