Skip job artifacts deleted event when jobs are destroyed
What does this MR do and why?
This MR stops publishing Ci::JobArtifactsDeletedEvent when job artifacts are destroyed with their pipeline or project. The change is behind a derisk feature flag that is disabled by default.
When a pipeline or a project is destroyed, Ci::JobArtifacts::DestroyAssociationsService destroys the job artifacts in batches of up to 100 through Ci::JobArtifacts::DestroyBatchService. For each batch, the EE after_batch_destroy_hook publishes one Ci::JobArtifactsDeletedEvent. The only subscriber is Security::Scans::PurgeByJobIdWorker. It sets security_scans.status to purged for the jobs in the event. Each batch enqueues one purge job, and the number of purge jobs grows with the number of pipelines that have artifacts.
On this path the jobs are destroyed too, and the purge has no effect:
security_scanshas loose foreign keys top_ci_builds(build_id) andprojects(project_id), bothasync_delete. Loose foreign key processing deletes the scans after the jobs.security_findingsandvulnerability_partial_scansreferencesecurity_scanswithON DELETE CASCADE. Their rows are deleted with each scan, whatever the scan status.- The
security_findingspartition detach checks the scan of the last finding in a partition. A scan that is not purged on this path can delay a detach only until loose foreign key processing deletes it.
This MR adds a jobs_destroyed: keyword to DestroyBatchService. Ci::JobArtifacts::DestroyAssociationsService sets it, because it is only called when the jobs are destroyed too. The EE hook does not publish the event when the keyword is set and the skip_job_artifacts_deleted_event_on_destroy feature flag is enabled for the project of the batch.
The other callers of DestroyBatchService keep publishing the event. These are DestroyAllExpiredService, BuildEraseService, DeleteService, BulkDeleteByProjectService and UpdateUnknownLockedStatusService. Their jobs stay in the database, and their scans still need the purge.
If a pipeline destroy fails after its artifacts are destroyed, its jobs stay in the database without the purge. The stale scan purge cron purges those scans later. This case is possible when ci_destroy_pipeline_artifacts_outside_transaction from !262274 (merged) is enabled. That MR changes the same pipeline destroy flow. It calls the same Ci::Pipelines::DestroyAssociationsService, which uses Ci::JobArtifacts::DestroyAssociationsService, and it changes no file that this MR changes.
Related to #633838.
Feature flag
| Flag | Type | Default | Actor |
|---|---|---|---|
skip_job_artifacts_deleted_event_on_destroy |
gitlab_com_derisk |
disabled | project of the first artifact in the batch |
When jobs_destroyed is set, the batch comes from one pipeline or one project. The batch service already preloads the project of each artifact, so the flag check adds no query to load the actor.
Rollout issue: #636324 (closed).
How to set up and validate locally
-
Enable the feature flag in the Rails console:
Feature.enable(:skip_job_artifacts_deleted_event_on_destroy) -
In a terminal, watch the Sidekiq log for the purge worker:
gdk tail rails-background-jobs | grep Security::Scans::PurgeByJobIdWorker -
In the Rails console, destroy a pipeline whose jobs have artifacts:
pipeline = Ci::JobArtifact.last.job.pipeline Ci::DestroyPipelineService.new(pipeline.project, pipeline.project.first_owner).execute(pipeline) -
Make sure that the log shows no
PurgeByJobIdWorkerjob. -
Disable the feature flag, and destroy another pipeline that has artifacts. The log shows one
PurgeByJobIdWorkerjob for each batch of artifacts.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.