Skip to content

Unlock non-successful pipelines

Erick Bajao requested to merge eb-unlock-non-successful-pipelines into master

What does this MR do and why?

Resolves #266958 (closed) and #387087 (closed)

Unlock non-successful pipelines

Unlocks previous pipelines on the ref even when the status of the latest is not successful.

This also keeps the latest successful pipeline locked.

How to set up and validate locally

You can clone this test project locally: https://gitlab.com/iamricecake/test-pipeline-unlock

This test project would make it easier for you to either fail or block the pipeline for testing purposes.

Steps:

  1. In rails console ensure that the feature is disabled for now 
    Feature.disable(:ci_unlock_non_successful_pipelines)
  2. First, run one successful pipeline, this is to test later that the latest successful pipeline is kept locked.
  3. Next, if you are using the test project, run a pipeline manually and force it to fail by setting the variable FAIL to 1.
    • Since the flag is still disabled, we expect both pipelines to be locked. Confirm this in the rails console by checking their locked attribute.
  4. Now, enable the feature 
    Feature.enable(:ci_unlock_non_successful_pipelines)
  5. Next, run another pipeline manually and force it to fail by setting the variable FAIL to 1.
  6. You can force enqueue the ci_schedule_unlock_pipelines_in_queue_worker cron job in the sidekiq admin.
  7. Go back to the rails console and check that the previous "failed" pipeline is now unlocked. But the previous successful pipeline should still be locked because it is the last successful pipeline in the ref.
  8. Take note of the latest failed pipeline's locked status. It should be locked.
  9. Now, we run another pipeline but this time we want to test the "blocked" scenario. Set the variable BLOCK to 1.
  10. You can again force enqueue the ci_schedule_unlock_pipelines_in_queue_worker cron job in the sidekiq admin.
  11. Confirm that the previous failed pipeline is now unlocked. But the previous successful pipeline should still be locked.
  12. Now we run another pipeline but don't set any variables, so we can test the unlocking of the previous successful pipeline.
  13. You can again force enqueue the ci_schedule_unlock_pipelines_in_queue_worker cron job in the sidekiq admin.
  14. Confirm that the previous successful pipeline is now unlocked. This is because we now have a new latest successful pipeline.

SQL queries

These are sample queries generated, gathered during spec execution:

I updated the ref and pipeline IDs to match the production test project (https://gitlab.com/iamricecake/test-pipeline-unlock) so I can test on database-lab.

app/services/ci/refs/enqueue_pipelines_to_unlock_service.rb:38 | excluded_ids = before_pipeline.same_family_pipeline_ids.map(&:id)

database-lab link

WITH RECURSIVE "base_and_descendants" AS ((SELECT "ci_pipelines"."id", "ci_pipelines"."ref", "ci_pipelines"."sha", "ci_pipelines"."before_sha", "ci_pipelines"."created_at", "ci_pipelines"."updated_at", "ci_pipelines"."tag", "ci_pipelines"."yaml_errors", "ci_pipelines"."committed_at", "ci_pipelines"."project_id", "ci_pipelines"."status", "ci_pipelines"."started_at", "ci_pipelines"."finished_at", "ci_pipelines"."duration", "ci_pipelines"."user_id", "ci_pipelines"."lock_version", "ci_pipelines"."auto_canceled_by_id", "ci_pipelines"."pipeline_schedule_id", "ci_pipelines"."source", "ci_pipelines"."config_source", "ci_pipelines"."protected", "ci_pipelines"."failure_reason", "ci_pipelines"."iid", "ci_pipelines"."merge_request_id", "ci_pipelines"."source_sha", "ci_pipelines"."target_sha", "ci_pipelines"."external_pull_request_id", "ci_pipelines"."ci_ref_id", "ci_pipelines"."locked", "ci_pipelines"."partition_id" FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 1049359408)
UNION
(SELECT "ci_pipelines"."id", "ci_pipelines"."ref", "ci_pipelines"."sha", "ci_pipelines"."before_sha", "ci_pipelines"."created_at", "ci_pipelines"."updated_at", "ci_pipelines"."tag", "ci_pipelines"."yaml_errors", "ci_pipelines"."committed_at", "ci_pipelines"."project_id", "ci_pipelines"."status", "ci_pipelines"."started_at", "ci_pipelines"."finished_at", "ci_pipelines"."duration", "ci_pipelines"."user_id", "ci_pipelines"."lock_version", "ci_pipelines"."auto_canceled_by_id", "ci_pipelines"."pipeline_schedule_id", "ci_pipelines"."source", "ci_pipelines"."config_source", "ci_pipelines"."protected", "ci_pipelines"."failure_reason", "ci_pipelines"."iid", "ci_pipelines"."merge_request_id", "ci_pipelines"."source_sha", "ci_pipelines"."target_sha", "ci_pipelines"."external_pull_request_id", "ci_pipelines"."ci_ref_id", "ci_pipelines"."locked", "ci_pipelines"."partition_id" FROM "ci_pipelines", "base_and_descendants", "ci_sources_pipelines" WHERE "ci_sources_pipelines"."pipeline_id" = "ci_pipelines"."id" AND "ci_sources_pipelines"."source_pipeline_id" = "base_and_descendants"."id" AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")) SELECT "id" FROM "base_and_descendants" AS "ci_pipelines"

--

app/models/ci/ref.rb:71:in 'last_successful_ci_source_pipeline'

database-lab link

SELECT "ci_pipelines"."id", "ci_pipelines"."ref", "ci_pipelines"."sha", "ci_pipelines"."before_sha", "ci_pipelines"."created_at", "ci_pipelines"."updated_at", "ci_pipelines"."tag", "ci_pipelines"."yaml_errors", "ci_pipelines"."committed_at", "ci_pipelines"."project_id", "ci_pipelines"."status", "ci_pipelines"."started_at", "ci_pipelines"."finished_at", "ci_pipelines"."duration", "ci_pipelines"."user_id", "ci_pipelines"."lock_version", "ci_pipelines"."auto_canceled_by_id", "ci_pipelines"."pipeline_schedule_id", "ci_pipelines"."source", "ci_pipelines"."config_source", "ci_pipelines"."protected", "ci_pipelines"."failure_reason", "ci_pipelines"."iid", "ci_pipelines"."merge_request_id", "ci_pipelines"."source_sha", "ci_pipelines"."target_sha", "ci_pipelines"."external_pull_request_id", "ci_pipelines"."ci_ref_id", "ci_pipelines"."locked", "ci_pipelines"."partition_id" FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 81032686 AND ("ci_pipelines"."source" IN (1, 2, 3, 4, 5, 6, 7, 8, 10, 11) OR "ci_pipelines"."source" IS NULL) AND ("ci_pipelines"."status" IN ('success')) ORDER BY "ci_pipelines"."id" DESC LIMIT 1

--

app/services/ci/refs/enqueue_pipelines_to_unlock_service.rb:46 | excluded_ids.concat(pipeline.same_family_pipeline_ids.map(&:id))

database-lab link

WITH RECURSIVE "base_and_descendants" AS ((SELECT "ci_pipelines"."id", "ci_pipelines"."ref", "ci_pipelines"."sha", "ci_pipelines"."before_sha", "ci_pipelines"."created_at", "ci_pipelines"."updated_at", "ci_pipelines"."tag", "ci_pipelines"."yaml_errors", "ci_pipelines"."committed_at", "ci_pipelines"."project_id", "ci_pipelines"."status", "ci_pipelines"."started_at", "ci_pipelines"."finished_at", "ci_pipelines"."duration", "ci_pipelines"."user_id", "ci_pipelines"."lock_version", "ci_pipelines"."auto_canceled_by_id", "ci_pipelines"."pipeline_schedule_id", "ci_pipelines"."source", "ci_pipelines"."config_source", "ci_pipelines"."protected", "ci_pipelines"."failure_reason", "ci_pipelines"."iid", "ci_pipelines"."merge_request_id", "ci_pipelines"."source_sha", "ci_pipelines"."target_sha", "ci_pipelines"."external_pull_request_id", "ci_pipelines"."ci_ref_id", "ci_pipelines"."locked", "ci_pipelines"."partition_id" FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 1047322538)
UNION
(SELECT "ci_pipelines"."id", "ci_pipelines"."ref", "ci_pipelines"."sha", "ci_pipelines"."before_sha", "ci_pipelines"."created_at", "ci_pipelines"."updated_at", "ci_pipelines"."tag", "ci_pipelines"."yaml_errors", "ci_pipelines"."committed_at", "ci_pipelines"."project_id", "ci_pipelines"."status", "ci_pipelines"."started_at", "ci_pipelines"."finished_at", "ci_pipelines"."duration", "ci_pipelines"."user_id", "ci_pipelines"."lock_version", "ci_pipelines"."auto_canceled_by_id", "ci_pipelines"."pipeline_schedule_id", "ci_pipelines"."source", "ci_pipelines"."config_source", "ci_pipelines"."protected", "ci_pipelines"."failure_reason", "ci_pipelines"."iid", "ci_pipelines"."merge_request_id", "ci_pipelines"."source_sha", "ci_pipelines"."target_sha", "ci_pipelines"."external_pull_request_id", "ci_pipelines"."ci_ref_id", "ci_pipelines"."locked", "ci_pipelines"."partition_id" FROM "ci_pipelines", "base_and_descendants", "ci_sources_pipelines" WHERE "ci_sources_pipelines"."pipeline_id" = "ci_pipelines"."id" AND "ci_sources_pipelines"."source_pipeline_id" = "base_and_descendants"."id" AND "ci_sources_pipelines"."source_project_id" = "ci_sources_pipelines"."project_id")) SELECT "id" FROM "base_and_descendants" AS "ci_pipelines"

--

app/models/concerns/each_batch.rb:62:in 'each_batch'

database-lab link

SELECT "ci_pipelines"."id" FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 81032686 AND "ci_pipelines"."locked" = 1 AND "ci_pipelines"."id" < 1049359408 AND "ci_pipelines"."id" NOT IN (1049359408, 1047322538) ORDER BY "ci_pipelines"."id" ASC LIMIT 1

--

app/models/concerns/each_batch.rb:81:in 'block in each_batch'

database-lab link

SELECT "ci_pipelines"."id" FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 81032686 AND "ci_pipelines"."locked" = 1 AND "ci_pipelines"."id" < 1049359408 AND "ci_pipelines"."id" NOT IN (1049359408, 1047322538) AND "ci_pipelines"."id" >= 1049357055 ORDER BY "ci_pipelines"."id" ASC LIMIT 1 OFFSET 2

--

app/services/ci/refs/enqueue_pipelines_to_unlock_service.rb:15:in 'block in execute' | pipeline_ids = batch.pluck(:id)

database-lab link

SELECT "ci_pipelines"."id" FROM "ci_pipelines" WHERE "ci_pipelines"."ci_ref_id" = 81032686 AND "ci_pipelines"."locked" = 1 AND "ci_pipelines"."id" < 1049359408 AND "ci_pipelines"."id" NOT IN (1049359408, 1047322538) AND "ci_pipelines"."id" >= 1049357055 AND "ci_pipelines"."id" < 1049359407

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Erick Bajao

Merge request reports