Coalesce concurrent security report ingestion per project
What does this MR do and why?
Changes Security::StoreSecurityReportsByProjectWorker from deduplicate :until_executing to :until_executed, if_deduplicated: :reschedule_once.
until_executing releases the deduplication lock when the job starts, so a second job for the same project can execute alongside the first. Both then contend on the same (project_id, fingerprint) rows in vulnerability_identifiers — which is exactly where the statement timeouts in #629530 land. The contention is self-inflicted.
Why coalescing is safe here
perform resolves the latest pipeline from the Redis register at run time rather than taking a pipeline id, so a coalesced duplicate loses no work — the surviving job picks up whatever the latest state is. reschedule_once still re-runs the job after the in-flight one finishes, so the newest result is produced, just without the two overlapping.
This is one of several contention changes identified in the RCA. The companion one is adding a concurrency_limit to this worker. An earlier version of this description also named Security::Scans::IngestReportsWorker as writing the same rows; that was wrong — it only enqueues Security::StoreScansWorker and never reaches the ingestion slice transaction. The other writer of that transaction is PackageMetadata::GlobalAdvisoryScanWorker via CVS, which already has deduplication and a concurrency limit.
There is no feature flag: a deduplication strategy is a class-level DSL setting with no runtime toggle, so the rollback is a revert.
References
- Closes #629991 (closed)
- RCA: #629530
- Parent epic: &23596
Screenshots or screen recordings
No UI change.
How to set up and validate locally
bundle exec rspec ee/spec/workers/security/store_security_reports_by_project_worker_spec.rbLocally: 33 examples, 0 failures. RuboCop clean.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.