Skip to content

Update BulkImports to handle empty relation status responses

What does this MR do and why?

GitLab Migration aka BulkImports requests relations export from source GitLab instance in order to download it and import its data on destination instance.

However, if the target instance requests the export relation's status before the export process had started, the relation status response is empty therefore GitLab Migration should not mark the pipeline as failed as it's currently doing, and instead if should wait for the export file generation to complete.

So this change modifies GitLab Migration to not consider empty relation status responses as a failed pipeline and instead checks if the relation's status is empty and re-enqueue the pipeline to test the relation's status again later.

Related to:

Race condition when migrating group and projects (#358743 - closed)

Group Import Error: NoMethodError: undefined me... (#352796 - closed)

Handle BulkImports::ExportStatus incorrect expo... (!83811 - merged)

Screenshots or screen recordings

RelationStatus

How to set up and validate locally

To test the race condition scenario, it's a bit tricky because we need to make the export file generation take longer than the execution of the pipeline job.

One option to do that in the local environment is to add a sleep in the service that generates the file

https://gitlab.com/gitlab-org/gitlab/blob/cd0296a7b9402b1bfcdb39c14a332144186a84a4/app/services/bulk_imports/relation_export_service.rb#L17-26

def execute
  sleep 100 if %w[self namespace_settings].include?(relation)
  
  find_or_create_export! do |export|
    remove_existing_export_file!(export)
    export_service.execute
    compress_exported_relation
    upload_compressed_file(export)
  end
ensure
  FileUtils.remove_entry(config.export_path)
end
  1. Feature.enable(:bulk_import).
  2. Create a top-level group
  3. Go to /groups/new#import-group-pane page and enter the instance URL and access token (needs to be api & read_repository scope).
  4. Make sure the group doesn't have previously exported files /api/v4/groups/:id/export_relations/status
  5. If the group has exported files, delete the files from the table bulk_import_exports
  6. Select the newly created group and click Import.
  7. Wait for Group import to complete and verify the imported group.
  8. Verify if all bulk_import_trackers have status 2 (finished)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports