Skip job artifacts deleted event when jobs are destroyed

What does this MR do and why?

This MR stops publishing Ci::JobArtifactsDeletedEvent when job artifacts are destroyed with their pipeline or project. The change is behind a derisk feature flag that is disabled by default.

When a pipeline or a project is destroyed, Ci::JobArtifacts::DestroyAssociationsService destroys the job artifacts in batches of up to 100 through Ci::JobArtifacts::DestroyBatchService. For each batch, the EE after_batch_destroy_hook publishes one Ci::JobArtifactsDeletedEvent. The only subscriber is Security::Scans::PurgeByJobIdWorker. It sets security_scans.status to purged for the jobs in the event. Each batch enqueues one purge job, and the number of purge jobs grows with the number of pipelines that have artifacts.

On this path the jobs are destroyed too, and the purge has no effect:

  • security_scans has loose foreign keys to p_ci_builds (build_id) and projects (project_id), both async_delete. Loose foreign key processing deletes the scans after the jobs.
  • security_findings and vulnerability_partial_scans reference security_scans with ON DELETE CASCADE. Their rows are deleted with each scan, whatever the scan status.
  • The security_findings partition detach checks the scan of the last finding in a partition. A scan that is not purged on this path can delay a detach only until loose foreign key processing deletes it.

This MR adds a jobs_destroyed: keyword to DestroyBatchService. Ci::JobArtifacts::DestroyAssociationsService sets it, because it is only called when the jobs are destroyed too. The EE hook does not publish the event when the keyword is set and the skip_job_artifacts_deleted_event_on_destroy feature flag is enabled for the project of the batch.

The other callers of DestroyBatchService keep publishing the event. These are DestroyAllExpiredService, BuildEraseService, DeleteService, BulkDeleteByProjectService and UpdateUnknownLockedStatusService. Their jobs stay in the database, and their scans still need the purge.

If a pipeline destroy fails after its artifacts are destroyed, its jobs stay in the database without the purge. The stale scan purge cron purges those scans later. This case is possible when ci_destroy_pipeline_artifacts_outside_transaction from !262274 (merged) is enabled. That MR changes the same pipeline destroy flow. It calls the same Ci::Pipelines::DestroyAssociationsService, which uses Ci::JobArtifacts::DestroyAssociationsService, and it changes no file that this MR changes.

Related to #633838.

Feature flag

Flag Type Default Actor
skip_job_artifacts_deleted_event_on_destroy gitlab_com_derisk disabled project of the first artifact in the batch

When jobs_destroyed is set, the batch comes from one pipeline or one project. The batch service already preloads the project of each artifact, so the flag check adds no query to load the actor.

Rollout issue: #636324 (closed).

How to set up and validate locally

  1. Enable the feature flag in the Rails console:

    Feature.enable(:skip_job_artifacts_deleted_event_on_destroy)
  2. In a terminal, watch the Sidekiq log for the purge worker:

    gdk tail rails-background-jobs | grep Security::Scans::PurgeByJobIdWorker
  3. In the Rails console, destroy a pipeline whose jobs have artifacts:

    pipeline = Ci::JobArtifact.last.job.pipeline
    Ci::DestroyPipelineService.new(pipeline.project, pipeline.project.first_owner).execute(pipeline)
  4. Make sure that the log shows no PurgeByJobIdWorker job.

  5. Disable the feature flag, and destroy another pipeline that has artifacts. The log shows one PurgeByJobIdWorker job for each batch of artifacts.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Gregory Havenga

Merge request reports

Loading
Loading