Loading
Add concurrency limit to Security::Scans::PurgeByJobIdWorker
What does this MR do and why?
This MR adds an explicit concurrency limit of 25 to Security::Scans::PurgeByJobIdWorker. It also removes duplicate job ids from the event before the worker calls Security::PurgeScansService.purge_by_build_ids.
The worker subscribes to Ci::JobArtifactsDeletedEvent. It had no explicit concurrency limit. On GitLab.com it got the default limit, which is 25% of the threads on its shard and far above 25. On self-managed and GitLab Dedicated the default calculation is behind the sidekiq_concurrency_limit_default_calculation ops flag, which is off by default, so the worker had no limit at all.
The artifact destroy path publishes one event for each batch of artifacts, and each event enqueues one purge job. A large destroy can therefore enqueue many jobs at once. Each job is cheap, but nothing bounded how many run at once. A burst can hold many threads and database connections on the shard, which also runs user facing workers such as MergeTrains::RefreshWorker. The related confidential issue has the details of the burst that prompted this change.
Why 25
The status update is idempotent. It sets security_scans.status to purged. The run also writes the shared purge cursor, which !262572 (merged) fixes.
When the builds are deleted, loose foreign keys delete the security_scans rows anyway. When only the artifacts expire or a job is erased, a delay postpones the purged status, the report purged alert and the partition detach. No data is lost.
A limit of 25 is the same as other Security EventStore subscribers that process delete or archive events, for example Security::AnalyzersStatus::ProcessDeletedEventsWorker.
We expect normal load to stay below 25 running jobs, so the limit does not trigger at normal load. We will confirm this from the concurrency limit metrics after the deploy. Jobs over the limit wait in the concurrency limit buffer, a Redis list in the Gitlab::Redis::ConcurrencyLimit store. That store falls back to the shared state Redis when it is not configured. While jobs are buffered, ConcurrencyLimit::ResumeWorker releases them only up to the limit of 25 running jobs. A backlog then drains at the pace of that cycle, and the purged status can lag after a burst.
The limit trades that buffer and delay for a bound on shard threads and database connections. If the buffer for this worker stays large at normal load, the disable_sidekiq_concurrency_limit_middleware_Security::Scans::PurgeByJobIdWorker feature flag stops new jobs from going into the buffer. Jobs that are already in the buffer still drain through ConcurrencyLimit::ResumeWorker. To avoid the burst itself, a follow-up can skip the purge when a project or pipeline is destroyed (see "Not in this MR").
Duplicate job ids
The publisher sends the job_id of each deleted artifact. A job can have more than one artifact, so the same id can occur more than once in one event. The worker now calls uniq on the ids. This MR does not change the publisher in ee/app/services/ee/ci/job_artifacts/destroy_batch_service.rb, because the CI group owns it.
Not in this MR
- The shared Redis cursor in
Security::PurgeScansService. !262572 (merged) fixes it. - Skipping the purge when a project or pipeline is destroyed. This touches the CI artifact destroy path, and we will coordinate it with the CI group.
References
Screenshots or screen recordings
Not applicable. This is a backend change with no UI.
How to set up and validate locally
-
Run the worker spec:
bundle exec rspec ee/spec/workers/security/scans/purge_by_job_id_worker_spec.rb -
In a Rails console, check the limit:
Security::Scans::PurgeByJobIdWorker.get_concurrency_limit # => 25
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Gregory Havenga