Update BulkImports to handle empty relation status responses
What does this MR do and why?
GitLab Migration aka BulkImports requests relations export from source GitLab instance in order to download it and import its data on destination instance.
However, if the target instance requests the export relation's status before the export process had started, the relation status response is empty therefore GitLab Migration should not mark the pipeline as failed as it's currently doing, and instead if should wait for the export file generation to complete.
So this change modifies GitLab Migration to not consider empty relation status responses as a failed pipeline and instead checks if the relation's status is empty and re-enqueue the pipeline to test the relation's status again later.
Related to:
Race condition when migrating group and projects (#358743 - closed)
Group Import Error: NoMethodError: undefined me... (#352796 - closed)
Handle BulkImports::ExportStatus incorrect expo... (!83811 - merged)
Screenshots or screen recordings
How to set up and validate locally
To test the race condition scenario, it's a bit tricky because we need to make the export file generation take longer than the execution of the pipeline job.
One option to do that in the local environment is to add a sleep in the service that generates the file
def execute
sleep 100 if %w[self namespace_settings].include?(relation)
find_or_create_export! do |export|
remove_existing_export_file!(export)
export_service.execute
compress_exported_relation
upload_compressed_file(export)
end
ensure
FileUtils.remove_entry(config.export_path)
end
-
Feature.enable(:bulk_import)
. - Create a top-level group
- Go to
/groups/new#import-group-pane
page and enter the instance URL and access token (needs to beapi
&read_repository
scope). - Make sure the group doesn't have previously exported files
/api/v4/groups/:id/export_relations/status
- If the group has exported files, delete the files from the table
bulk_import_exports
- Select the newly created group and click Import.
- Wait for Group import to complete and verify the imported group.
- Verify if all
bulk_import_trackers
have status 2 (finished)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.