Skip to content

Keep pipelines with artifacts_locked

Nicholas Wittstruckrequested to merge
gitlab-community/gitlab-org/gitlab:nwittstruck-pipeline-removal-artifacts-locked into master

What does this MR do and why?

Currently the retention of pipelines does not take into account if it is :artifacts_locked. This can be quite surprising for users, as this will delete artifacts regardless of the configuration.

Before this MR, :artifacts_locked only had an effect on the artifact retention for the cleanup job that uses expire_in on artifacts, but not on the cleanup job for pipelines. This MR applies the setting also to pipeline retention.

This MR introduces a feature flag named :ci_skip_locked_pipelines

🛠️ with ❤️ at Siemens

References

How to set up and validate locally

If you have an existing project in your GDK with older pipelines, you can start there. Gitlab-org/gitlab-shell is a good example case that has artifacts and pipelines. Otherwise, you can also create a new project and pipelines like this:

  1. Enable feature flag via rails c:

Setup project / pipelines

  1. Create a new project
  2. Configure a pipeline that creates an artifact: Build > Pipeline editor:
stages:
- build

build-job:
stage: build
script:
  - echo 'Hello World' >> artifact.txt
artifacts:
  paths:
    - artifact.txt
  1. Configure automatic pipeline cleanup Settings > CI/CD > General pipelines > Automatic pipeline cleanup to 1 day.
  2. Make sure artifacts are supposed to be kept: Settings > CI/CD > Artifacts > Keep artifacts from most recent successful job.
  3. Now run this pipeline at least twice. Make sure that one pipeline is :artifacts_locked and the other one is :unlocked with Ci::Pipeline.find(<id>).locked)

Test

Feature.enable(:labels_archive)

If you've created new pipelines, you'll need to change the date on them. Otherwise they won't be caught by the removal cron, unless you wait a day:

Ci::Pipeline.find(<id>).update(created_at: DateTime.now - 2.days)

You can wait for the scheduler to run at some point, but you can also trigger it like this:

Ci::ScheduleOldPipelinesRemovalCronWorker.new.perform
Ci::DestroyOldPipelinesWorker.new.perform_work

You might have to call Ci::DestroyOldPipelinesWorker.new.perform_work a few times depending if you have the expiration configured on other projects as well. It might take a few runs until it reaches your test project.

Database Query

Query Plan for `.explain` called on Ci::DestroyOldPipelinesWorker.new.perform_work > pipelines 
EXPLAIN SELECT "p_ci_pipelines".* FROM "p_ci_pipelines" WHERE "p_ci_pipelines"."project_id" = 19 AND "p_ci_pipelines"."created_at" < '2025-07-09 17:14:13.706268' AND "p_ci_pipelines"."locked" = 0 LIMIT 250

                                                                         QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.15..1.63 rows=3 width=366)
   ->  Append  (cost=0.15..1.63 rows=3 width=366)
         ->  Index Scan using ci_pipelines_100_project_id_status_config_source_idx on ci_pipelines_100 p_ci_pipelines_1  (cost=0.15..1.62 rows=1 width=339)
               Index Cond: (project_id = 19)
               Filter: ((created_at < '2025-07-09 17:14:13.706268'::timestamp without time zone) AND (locked = 0))
         ->  Seq Scan on ci_pipelines_101 p_ci_pipelines_2  (cost=0.00..0.00 rows=1 width=380)
               Filter: ((created_at < '2025-07-09 17:14:13.706268'::timestamp without time zone) AND (project_id = 19) AND (locked = 0))
         ->  Seq Scan on ci_pipelines_102 p_ci_pipelines_3  (cost=0.00..0.00 rows=1 width=380)
               Filter: ((created_at < '2025-07-09 17:14:13.706268'::timestamp without time zone) AND (project_id = 19) AND (locked = 0))
(9 rows)

MR acceptance checklist

MR Checklist ( @nwittstruck)
Edited by Nicholas Wittstruck

Merge request reports