Let a StoreScansWorker retry take back its own store lease
What does this MR do and why?
Security::StoreGroupedScansService takes a store lease for each pipeline and report type. Since !262568 (merged), the service raises LeaseTakenError when the lease is taken, and Security::StoreScansWorker retries with a backoff.
The full backoff is longer than the 30 minute lease TTL, so a job that starts its retries against a lease waits it out. A job can still lose its scans, though. When a run of the worker is interrupted, for example when its Sidekiq process is killed, it does not release the lease, and Sidekiq runs the same job again. If the job has already used two or more of its retries, for example waiting for another job's lease, its remaining backoff is shorter than the lease TTL. Its last retries then fail with LeaseTakenError against the lease that the job itself still holds, and the scans of the pipeline are not stored.
This MR adds the store_scans_reentrant_lease feature flag. When the flag is on:
- The worker gives its Sidekiq job ID to
Security::StoreScansService, which gives it toSecurity::StoreGroupedScansServiceandSecurity::StoreGroupedSbomScansService. - The service uses the job ID as the lease UUID. The job ID does not change between retries of a job.
- When the lease is free, the service takes it. When the lease exists and its UUID is the job ID, the service renews the lease and continues.
- When a different job holds the lease, the service still raises
LeaseTakenError. - The service renews the lease after it stores each artifact, so that a long run keeps its lease.
When the flag is off, or when no job ID is given, the service uses in_lock as before.
Design decision
This change is in the store service only. It uses Gitlab::ExclusiveLease directly, because Gitlab::ExclusiveLeaseHelpers#in_lock always makes a random UUID.
A shared option on in_lock (for example, reentrant_uuid:) can help other services that have the same problem. Many services call in_lock, so this MR does not change it. A follow-up can add the option and move this service to it.
The flagged path does not record the Gitlab::Instrumentation::ExclusiveLock wait and hold durations that in_lock records.
A lease with our job ID does not always mean that the earlier run has stopped. Sidekiq can requeue a job while its earlier run is still alive, for example when a process misses its reliable fetch heartbeat. With the flag on, both runs then store the scans of the pipeline at the same time, where before the second run backed off. The same overlap is already possible today once the lease expires.
When two runs overlap:
- Both runs use the same
Security::Scanrecords. If a scan already has findings, the other run does not store findings for that scan. It runs the deduplication again and marks the scan as succeeded, even when the first run is still storing findings. - If both runs insert the same findings at the same time, the second insert raises
ActiveRecord::RecordNotUnique.Security::StoreScanServicethen marks the scan aspreparation_failedwith aScanIngestionError. The last status that is written is kept. The scan can show aspreparation_failedwhen all of its findings are stored. If the other run then marks the scan as succeeded, theScanIngestionErrorstays in the processing errors of the scan. - The run that finishes first cancels the lease while the other run still stores scans. Another
Security::StoreScansWorkerjob for the same pipeline can then take the lease and store scans at the same time.
This MR does not fix the cause of the interruptions. A job that is interrupted on every run still fails, once it reaches the reliable fetch limit for interrupted jobs (max_retries_after_interruption) instead of on LeaseTakenError.
Feature flag and rollout
| Item | Value |
|---|---|
| Name | store_scans_reentrant_lease |
| Type | gitlab_com_derisk |
| Default | Off |
| Actor | Project of the pipeline |
| Rollout issue | #637028 |
Rollout plan:
- Enable the flag for a small number of projects. Make sure that fewer
StoreScansWorkerjobs die withLeaseTakenError, and that otherStoreScansWorkererrors do not increase. - Enable the flag for a percentage of projects, then for all projects.
- Remove the flag in a follow-up MR.
To disable the new behaviour, disable the flag. The service then uses in_lock again.
How to set up and validate locally
-
Run the specs:
bundle exec rspec \ ee/spec/services/security/store_grouped_scans_service_spec.rb \ ee/spec/services/security/store_grouped_sbom_scans_service_spec.rb \ ee/spec/services/security/store_scans_service_spec.rb \ ee/spec/workers/security/store_scans_worker_spec.rb -
Optional: in a Rails console, hold a lease with a job ID, then run the service with the same job ID:
Feature.enable(:store_scans_reentrant_lease) pipeline = Ci::Pipeline.find(<pipeline_id>) artifacts = pipeline.job_artifacts.sast.to_a key = "store_grouped_scans:#{pipeline.id}:sast" Gitlab::ExclusiveLease.new(key, timeout: 1.minute, uuid: 'job-a').try_obtain # Same job ID: the service renews the lease and stores the scans. Security::StoreGroupedScansService.execute(artifacts, pipeline, 'sast', lease_owner: 'job-a') Gitlab::ExclusiveLease.new(key, timeout: 1.minute, uuid: 'job-a').try_obtain # Different job ID: the service raises LeaseTakenError. Security::StoreGroupedScansService.execute(artifacts, pipeline, 'sast', lease_owner: 'job-b')
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
Related issues
- Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/636006
- Lease backoff: !262568 (merged)