Let a StoreScansWorker retry take back its own store lease

What does this MR do and why?

Security::StoreGroupedScansService takes a store lease for each pipeline and report type. Since !262568 (merged), the service raises LeaseTakenError when the lease is taken, and Security::StoreScansWorker retries with a backoff.

The full backoff is longer than the 30 minute lease TTL, so a job that starts its retries against a lease waits it out. A job can still lose its scans, though. When a run of the worker is interrupted, for example when its Sidekiq process is killed, it does not release the lease, and Sidekiq runs the same job again. If the job has already used two or more of its retries, for example waiting for another job's lease, its remaining backoff is shorter than the lease TTL. Its last retries then fail with LeaseTakenError against the lease that the job itself still holds, and the scans of the pipeline are not stored.

This MR adds the store_scans_reentrant_lease feature flag. When the flag is on:

  1. The worker gives its Sidekiq job ID to Security::StoreScansService, which gives it to Security::StoreGroupedScansService and Security::StoreGroupedSbomScansService.
  2. The service uses the job ID as the lease UUID. The job ID does not change between retries of a job.
  3. When the lease is free, the service takes it. When the lease exists and its UUID is the job ID, the service renews the lease and continues.
  4. When a different job holds the lease, the service still raises LeaseTakenError.
  5. The service renews the lease after it stores each artifact, so that a long run keeps its lease.

When the flag is off, or when no job ID is given, the service uses in_lock as before.

Design decision

This change is in the store service only. It uses Gitlab::ExclusiveLease directly, because Gitlab::ExclusiveLeaseHelpers#in_lock always makes a random UUID.

A shared option on in_lock (for example, reentrant_uuid:) can help other services that have the same problem. Many services call in_lock, so this MR does not change it. A follow-up can add the option and move this service to it.

The flagged path does not record the Gitlab::Instrumentation::ExclusiveLock wait and hold durations that in_lock records.

A lease with our job ID does not always mean that the earlier run has stopped. Sidekiq can requeue a job while its earlier run is still alive, for example when a process misses its reliable fetch heartbeat. With the flag on, both runs then store the scans of the pipeline at the same time, where before the second run backed off. The same overlap is already possible today once the lease expires.

When two runs overlap:

  • Both runs use the same Security::Scan records. If a scan already has findings, the other run does not store findings for that scan. It runs the deduplication again and marks the scan as succeeded, even when the first run is still storing findings.
  • If both runs insert the same findings at the same time, the second insert raises ActiveRecord::RecordNotUnique. Security::StoreScanService then marks the scan as preparation_failed with a ScanIngestionError. The last status that is written is kept. The scan can show as preparation_failed when all of its findings are stored. If the other run then marks the scan as succeeded, the ScanIngestionError stays in the processing errors of the scan.
  • The run that finishes first cancels the lease while the other run still stores scans. Another Security::StoreScansWorker job for the same pipeline can then take the lease and store scans at the same time.

This MR does not fix the cause of the interruptions. A job that is interrupted on every run still fails, once it reaches the reliable fetch limit for interrupted jobs (max_retries_after_interruption) instead of on LeaseTakenError.

Feature flag and rollout

Item Value
Name store_scans_reentrant_lease
Type gitlab_com_derisk
Default Off
Actor Project of the pipeline
Rollout issue #637028

Rollout plan:

  1. Enable the flag for a small number of projects. Make sure that fewer StoreScansWorker jobs die with LeaseTakenError, and that other StoreScansWorker errors do not increase.
  2. Enable the flag for a percentage of projects, then for all projects.
  3. Remove the flag in a follow-up MR.

To disable the new behaviour, disable the flag. The service then uses in_lock again.

How to set up and validate locally

  1. Run the specs:

    bundle exec rspec \
      ee/spec/services/security/store_grouped_scans_service_spec.rb \
      ee/spec/services/security/store_grouped_sbom_scans_service_spec.rb \
      ee/spec/services/security/store_scans_service_spec.rb \
      ee/spec/workers/security/store_scans_worker_spec.rb
  2. Optional: in a Rails console, hold a lease with a job ID, then run the service with the same job ID:

    Feature.enable(:store_scans_reentrant_lease)
    
    pipeline = Ci::Pipeline.find(<pipeline_id>)
    artifacts = pipeline.job_artifacts.sast.to_a
    key = "store_grouped_scans:#{pipeline.id}:sast"
    
    Gitlab::ExclusiveLease.new(key, timeout: 1.minute, uuid: 'job-a').try_obtain
    
    # Same job ID: the service renews the lease and stores the scans.
    Security::StoreGroupedScansService.execute(artifacts, pipeline, 'sast', lease_owner: 'job-a')
    
    Gitlab::ExclusiveLease.new(key, timeout: 1.minute, uuid: 'job-a').try_obtain
    
    # Different job ID: the service raises LeaseTakenError.
    Security::StoreGroupedScansService.execute(artifacts, pipeline, 'sast', lease_owner: 'job-b')

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Gregory Havenga

Merge request reports

Loading
Loading