Elasticsearch malware field on the SBOM occurrence_ref index goes stale when an advisory is withdrawn

Summary

The malware field on the SBOM occurrence_ref Elasticsearch index is only written during SBOM ingestion. Nothing refreshes it when the underlying malware data changes, so when an advisory is withdrawn the indexed value stays true until the project happens to run another pipeline.

Where

ee/lib/search/elastic/references/sbom/occurrence_ref.rb indexes the value in add_fields_from_associations:

malware: occurrence.malware_status

It is mapped as a boolean in ee/lib/search/elastic/types/sbom/occurrence_ref.rb and is filterable through by_malware in ee/lib/search/elastic/sbom_occurrence_ref_filters.rb, so a stale value is not inert — it changes which occurrences a malware filter returns.

Problem

There are two compounding defects. Adding a reindex trigger alone will not fix this.

1. Nothing reindexes an occurrence_ref when its malware status changes

Sbom::Ingestion::TrackOccurrenceRefsEsService is the only thing that tracks these documents for update, and its single caller is Sbom::Ingestion::IngestReportSliceService. So the document is written when a pipeline ingests an SBOM report and at no other time.

There is no reindex hook on Vulnerability create, update or delete, none on Sbom::OccurrencesVulnerability, and none on advisory withdrawal. The indexed value is a snapshot taken at ingestion.

2. malware_status would not change even if a reindex fired

This is not true anymore, Sbom::Occurrence#malware_status is fetched using MalwareAdvisoriesPreloader which returns false for withdrawn advisories.

Sbom::Occurrence#malware_status calls ::Vulnerability.malware_status_for(vulnerabilities), which is:

Array(vulnerabilities).any?(&:has_glam_identifier?)

There is no vulnerability state filter. The planned remediation for withdrawal in #612099 is to resolve the affected vulnerabilities, not delete them, and resolving does not remove the GLAM- entry from vulnerability_read.identifier_names. So a correctly triggered reindex would still write malware: true.

The value only flips to false if the vulnerability record or the sbom_occurrences_vulnerabilities join row is deleted, or the GLAM identifier is stripped from the read.

Note that Sbom::Occurrence already has a with_active_vulnerabilities scope filtering on ACTIVE_STATES, so the codebase has the notion of state-aware vulnerability queries. It is simply not applied on this path.

Impact

Index contains stale malware value, which means malware filters on the Dependency List return stale data.

Proposal

Both halves need addressing:

  1. Make the indexed value withdrawal-aware. Either make malware_status state-aware, or derive the indexed field from something that reflects withdrawal directly. This interacts with #612090 (closed) and #612099, so it is worth settling the intended semantics of malware_status first.
  2. Reindex affected occurrence_refs when malware status changes, rather than waiting for the next SBOM ingestion. Withdrawal is a bulk event affecting every occurrence of the advisory's packages across all projects, so this likely needs a batched backfill rather than a per-record callback.

Acceptance criteria

  • Withdrawing a malware advisory results in the indexed malware value becoming false for affected occurrence_refs, without requiring a new pipeline run.
  • The malware filter (by_malware) stops returning those occurrences.
  • The intended semantics of malware_status with respect to vulnerability state and advisory withdrawal are written down and applied consistently with the REST and GraphQL dependency surfaces.
  • Spec covering an advisory withdrawn after ingestion, asserting the indexed value is corrected.

Related

Edited by Rushik Subba