Coalesce concurrent security report ingestion per project

What does this MR do and why?

Changes Security::StoreSecurityReportsByProjectWorker from deduplicate :until_executing to :until_executed, if_deduplicated: :reschedule_once.

until_executing releases the deduplication lock when the job starts, so a second job for the same project can execute alongside the first. Both then contend on the same (project_id, fingerprint) rows in vulnerability_identifiers — which is exactly where the statement timeouts in #629530 land. The contention is self-inflicted.

Why coalescing is safe here

perform resolves the latest pipeline from the Redis register at run time rather than taking a pipeline id, so a coalesced duplicate loses no work — the surviving job picks up whatever the latest state is. reschedule_once still re-runs the job after the in-flight one finishes, so the newest result is produced, just without the two overlapping.

This is one of several contention changes identified in the RCA. The companion one is adding a concurrency_limit to this worker. An earlier version of this description also named Security::Scans::IngestReportsWorker as writing the same rows; that was wrong — it only enqueues Security::StoreScansWorker and never reaches the ingestion slice transaction. The other writer of that transaction is PackageMetadata::GlobalAdvisoryScanWorker via CVS, which already has deduplication and a concurrency limit.

There is no feature flag: a deduplication strategy is a class-level DSL setting with no runtime toggle, so the rollback is a revert.

References

Screenshots or screen recordings

No UI change.

How to set up and validate locally

bundle exec rspec ee/spec/workers/security/store_security_reports_by_project_worker_spec.rb

Locally: 33 examples, 0 failures. RuboCop clean.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Bala Kumar

Merge request reports

Loading
Loading