Implement redis queue for unlocking pipelines

What does this MR do and why?

Implement redis queue for unlocking pipelines

We added a new redis queue to be able to unlock pipelines in a controlled manner instead of unlocking all of them in one go.

Related to #406255 (closed).

DB Queries Information

Update query generated by

unlocked_count = Ci::JobArtifact.where(id: job_artifacts.pluck(:id)).update_all(locked: :unlocked)

RAW SQL of updating 100 job artifacts locked status: https://paste.depesz.com/s/Aa

Full execution plan based on database-lab: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/22417/commands/72339

Update query generated by

@unlocked_pipeline_artifacts_count = pipeline.pipeline_artifacts.update_all(locked: :unlocked)

RAW SQL:

UPDATE "ci_pipeline_artifacts" SET "locked" = 0 WHERE "ci_pipeline_artifacts"."pipeline_id" = 977660488

Full exec plan based on database-lab: https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/22417/commands/72340

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

1. In rails console enable the feature flag:

   > Feature.enable(:ci_unlock_pipelines_queue) # This enables the behavior of enqueuing the pipelines to be unlocked

2. Clone this sample project locally: https://gitlab.com/iamricecake/test-pipeline-unlock
   • By default, it should have enabled "Keep artifacts from most recent succesful jobs" under "Settings > CI/CD > Artifacts".
3. Run a pipeline and wait for it to complete.
   • The artifacts generated on the sample project are set to expire in 1 minute.
   • After a minute, you can check the most recent job details page, and it will show the message "These artifacts are the latest. They will not be deleted (even if expired) until newer artifacts are available."
4. Run another pipeline and wait to complete.
5. Confirm that we now have enqueued an unlock pipeline request through the rails console:

   > Ci::UnlockPipelineRequest.total_pending
   => 1

6. Now we enable the feature flag for the limited capacity worker which will process the queue:

   > Feature.enable(:ci_unlock_pipelines) # This is for the limited capacity worker that processes the queue which defaults to LOW concurrency

7. Then let's manually enqueue the cron job through http://gdk.test:3000/admin/sidekiq/cron.
   • Look for ci_schedule_unlock_pipelines_worker and click on Enqueue Now.
   • You can run gdk tail rails-background-jobs | grep UnlockPipeline to check when the unlock task has executed.
8. Confirm that there are no more pending unlock requests:

   > Ci::UnlockPipelineRequest.total_pending
   => 0

9. Go visit the job details page of the older job and see the message "The artifacts were removed x minutes ago".

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.