Destroy job artifacts outside the pipeline destroy transaction
What does this MR do and why?
Shortens the ci database transaction that Ci::DestroyPipelineService opens when it destroys a pipeline.
Ci::Pipeline has a before_destroy callback that deletes all of the pipeline's job artifacts in batches: 100 builds at a time, then 100 artifacts at a time. Because it is a before_destroy callback, the whole cleanup runs inside the pipeline.destroy! transaction. For a large pipeline, that transaction holds a ci primary connection for the entire cleanup, including:
- main database queries (
with_destroy_preloadsloads each artifact batch's project, route, statistics and build artifacts size refresh) - Redis calls (Geo delete events and
Ci::JobArtifactsDeletedEvent, both already wrapped inskipping_transaction_check)
This MR adds Ci::Pipeline#destroy_with_job_artifacts_first!, which runs the same Ci::Pipelines::DestroyAssociationsService without an open transaction and then calls destroy! with the callback skipped for that call only. Ci::DestroyPipelineService uses it in place of pipeline.destroy!. Each artifact batch still commits its own short transaction (Ci::DeletedObject.bulk_import and the DELETE). The pipeline transaction is left with only the cascading DELETE and the trace chunk cleanup.
The cleanup reads from the ci primary. Inside the old transaction its reads went to the primary anyway; without it, a lagging replica could miss an artifact uploaded just before the delete.
If destroy! raises ActiveRecord::StaleObjectError, the retry runs the cleanup again before destroying, as the callback did before. This catches artifacts uploaded during the first attempt.
The callback stays in place, so destroy calls that don't go through the service still clean up artifacts.
The change is behind the ci_destroy_pipeline_artifacts_outside_transaction derisk flag, off by default.
Measurements and tradeoffs
Measured locally in the test environment, destroying 3 pipelines with 1,000 builds and 2,000 job artifacts each, timing every top-level ci transaction:
| Flag off | Flag on | |
|---|---|---|
| Longest ci transaction | 0.857s | 0.070s |
| Total time with a ci transaction open | 2.38s | 0.45s |
| Number of ci transactions | 3 | 63 (each 13ms or less) |
| ci queries inside the pipeline destroy transactions | 345 | 12 |
| main queries inside a ci transaction | 240 | 0 |
Ci::Pipelines::DestroyAssociationsService#destroy_records took 2.20s of the 2.38s. The test database is local and Geo is off, so production transactions will be longer than these timings. The query counts are the numbers that carry over.
Tradeoffs:
- Not atomic. If
pipeline.destroy!fails after the artifacts are deleted, the pipeline remains without its artifacts. Retrying the deletion finishes the job. - Wider race window. An artifact created between the cleanup and
pipeline.destroy!is removed by the foreign key cascade without aci_deleted_objectsrow, so its stored file is not deleted. The same race exists today, but the window is now the length of the cleanup rather than the end of the transaction. Jobs are cancelled before the cleanup starts. - Statistics update earlier. Project statistics are updated as soon as the artifacts are deleted, rather than after the pipeline transaction commits.
References
- Feature flag rollout issue: #635938
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
-
In the Rails console, enable the feature flag:
Feature.enable(:ci_destroy_pipeline_artifacts_outside_transaction) -
Pick a pipeline whose jobs have artifacts, and log each ci transaction's duration while destroying it:
pipeline = Ci::JobArtifact.last.job.pipeline sub = ActiveSupport::Notifications.subscribe('transaction.active_record') do |event| next unless Gitlab::Database.db_config_name(event.payload[:connection]) == 'ci' puts format('%.1fms %s', event.duration, event.payload[:outcome]) end Ci::DestroyPipelineService.new(pipeline.project, pipeline.project.owner).unsafe_execute(pipeline) ActiveSupport::Notifications.unsubscribe(sub) -
With the flag enabled, you see one short transaction per artifact batch and a final short transaction for the pipeline. Disable the flag and repeat with another pipeline: there is a single transaction that lasts for the whole artifact cleanup.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.