Skip to content

Resolve "Feature Request - Vulnerabilities Scan result API available before pipeline completes."

Schmil Mondererrequested to merge
513326-feature-request-vulnerabilities-scan-result-api-available-before-pipeline-completes into master

What does this MR do and why?

This change introduces a new approach for processing security scan reports in GitLab's CI/CD system. Instead of waiting for an entire pipeline to complete before processing security reports, the system now processes them as soon as all security jobs have completed.

The key improvements include:

  1. Faster processing: Security reports are now ingested immediately once all security jobs complete, rather than waiting for the whole pipeline to finish. This reduces the time between job completion and security findings being available.
  2. Event-driven architecture: A new event system publishes notifications when security jobs complete, triggering report processing through a dedicated background worker.
  3. Feature flag protection: The new behavior is controlled by a feature flag (ingest_sec_reports_when_sec_jobs_completed) that's disabled by default, allowing for safe rollout and easy rollback if needed.
  4. Improved efficiency: The system uses locks to prevent duplicate processing and only processes reports when all security jobs in a pipeline are complete and haven't been processed yet.
  5. Backward compatibility: The old pipeline-level processing remains as a fallback when the feature flag is disabled.

This change should make security scan results available to developers more quickly, improving the feedback loop for security issues in their code.

How to set up and validate locally

  1. Create a project with at least two security scan jobs
  2. To ensure that the security findings are available before the pipeline completes, add a final job with sleep
deploy:
  stage: deploy
  allow_failure: false
  script: 
    - echo "Deployed to production!"
    - sleep 30
  1. Run the pipeline and check that the security findings are available before the pipeline completes

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #513326

Edited by Schmil Monderer

Merge request reports