Skip to content

Race condition causes SBoM ingestion to be dropped when security jobs complete before pipeline

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

There's a race condition in the security report ingestion process that causes SBoM (Software Bill of Materials) jobs to be quietly dropped, resulting in empty dependency lists even when dependency scanning jobs complete successfully.

Steps to reproduce

Since this is a race condition, it cannot be reproduced with 100% reliability.

  1. Configure a project with dependency scanning that generates SBoM reports
  2. Insert a very-long running job after the dependency scanning job in order to keep the pipeline in running state.
  3. Check the dependency list - it will be empty despite successful dependency scanning jobs

Current behavior

SBoM reports are not being ingested into the dependency list, even when dependency scanning jobs complete successfully and generate valid SBoM artifacts.

Expected behavior

SBoM reports should be reliably ingested and displayed in the dependency list regardless of job completion timing.

Root Cause

The issue stems from a race condition introduced by the change to ingest vulnerabilities when security jobs finish (instead of when the pipeline finishes):

  1. Vulnerabilities are now ingested when all security jobs finish instead of when the pipeline finishes (introduced in MR !195012)
  2. SBoM ingestion is queued after security reports are stored (code reference)
  3. Sbom::ScheduleIngestReportsService still expects the pipeline to be completed (code reference)

This creates a race condition where SBoM ingestion only happens if the pipeline finishes before Sbom::ScheduleIngestReportsService is executed.

Impact

  • Reliability: Dependency lists are unreliably populated
  • Customer Impact: Multiple customers affected with code freezes due to missing security findings
  • User Experience: Users see empty dependency lists despite successful scans

Possible fixes

  1. Short-term: Consider disabling the ingest_sec_reports_when_sec_jobs_completed feature flag until this race condition is resolved
  2. Long-term: Update Sbom::ScheduleIngestReportsService to work with the new job-completion-based ingestion model instead of requiring pipeline completion

Related Issues

This issue is impacting multiple customer projects where dependency scanning jobs complete successfully but dependency lists remain empty.

Edited by 🤖 GitLab Bot 🤖