Add minimum and maximum source versions in BulkImport trackers config
What does this MR do and why?
GitLab Importer aka BulkImport uses a configuration file to define which pipelines will be executed when importing groups or projects.
However, some pipelines can't be executed when importing from sources instances running a GitLab version incompatible with the pipeline.
This change introduces the attributes minimum_source_version and maximum_source_version to the pipeline configuration to skip a pipeline depending on the source instance versions.
When the source instance version doesn't meet the version requirements, the pipeline trackers are created as skipped, which prevents the pipeline from being executed.
One example of when this situation can happen is in the pipeline BulkImports::Groups::Pipelines::NamespaceSettingsPipeline
that was added in GitLab 15.0
in !85128 (merged) and that depends on the export file namespace_settings
also added in the same version. An attempt to use GitLab Importer to import from a GitLab source instance using a version before 15.0 will make the pipeline wait for the namespace_settings
export file for a long time until the pipeline times out because the source instance won't generate the export file and consequently won't return any status of the export file in the endpoint (groups|projects)/:id/export_relations/status
.
Related to: #358743 (closed)
For the reviewers
- The method
BulkImports::Entity#create_pipeline_trackers!
was moved to a service (BulkImports::CreatePipelineTrackersService
) - BulkImports::Groups::Stage and BulkImports::Projects::Stage specs were written so that the configuration of the pipelines doesn't need to be duplicated in the specs
- Now,
BulkImports::Stage#pipelines
returns an array of hashes instead of an array of arrays, and because of that some classes and specs were updated to work with the new method response
Before
BulkImports::Stage#pipeline
[
[0, BulkImports::Groups::Pipelines::GroupPipeline],
[1, BulkImports::Groups::Pipelines::GroupAttributesPipeline]
...
]
After
BulkImports::Stage#pipeline
[
{ stage: 0, pipeline: BulkImports::Groups::Pipelines::GroupPipeline },
{ stage: 1, pipeline: BulkImports::Groups::Pipelines::GroupAttributesPipeline }
...
]
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
How to set up and validate locally
Testing if the pipelines are being skipped is tricky as we need to import from a source instance running an old GitLab version. One option to test in the local environment is to run a second GDK version and change the GitLab version.
Another option, that is easier, is to update the pipeline configuration as shown in the screen recording above
-
Feature.enable(:bulk_import)
. - Create a top-level group
- Go to
/groups/new#import-group-pane
page and enter the instance URL and access token (needs to beapi
&read_repository
scope). - Select the newly created group and click Import.
- Wait for Group import to complete and verify the imported group.
- Verify if the incompatible
bulk_import_trackers
have status -2 (skipped) in the DB
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.