Skip to content

Move CI pipelines migration to a lower stage during Gitlab Migration

What does this MR do and why?

Mentions #358776 (closed)

This MR:

  • Fixes exception during Project Migration when using Gitlab Migration PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "index_merge_requests_on_target_project_id_and_iid" DETAIL: Key (target_project_id, iid)=(697, 6) already exists.
  • Moves BulkImports::Projects::Pipelines::CiPipelinesPipeline from stage 4 to stage 5 in order to remove this potential race condition

This was happening due to merge requests & ci_pipelines migrations were in one stage (4), which could lead to 2 sidekiq jobs running at the same time. CI Pipelines reference MRs and create them during the import if they do not exist. IF CI Pipelines migration created one MR and then MRs migration tries to create the same MR - the exception occurs.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

  1. Feature.enable(:bulk_import).
  2. Feature.enable(:bulk_import_projects).
  3. Create a top level group and project in it.
  4. Create an MR and CI pipeline that references that MR
  5. Go to /groups/new#import-group-pane page and enter instance url and access token (needs to be api & read_repository scope).
  6. Select newly created group and click Import.
  7. Wait for Group import to complete and verify imported project.
  8. Verify that MR and CI pipelines are migrated
  9. Check bulk_import_failures table and verify no PG::UniqueViolation exception is raised & captured

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports