Skip to content

Add minimum and maximum source versions in BulkImport trackers config

What does this MR do and why?

GitLab Importer aka BulkImport uses a configuration file to define which pipelines will be executed when importing groups or projects.

However, some pipelines can't be executed when importing from sources instances running a GitLab version incompatible with the pipeline.

This change introduces the attributes minimum_source_version and maximum_source_version to the pipeline configuration to skip a pipeline depending on the source instance versions.

When the source instance version doesn't meet the version requirements, the pipeline trackers are created as skipped, which prevents the pipeline from being executed.

One example of when this situation can happen is in the pipeline BulkImports::Groups::Pipelines::NamespaceSettingsPipeline that was added in GitLab 15.0 in !85128 (merged) and that depends on the export file namespace_settings also added in the same version. An attempt to use GitLab Importer to import from a GitLab source instance using a version before 15.0 will make the pipeline wait for the namespace_settings export file for a long time until the pipeline times out because the source instance won't generate the export file and consequently won't return any status of the export file in the endpoint (groups|projects)/:id/export_relations/status.

Related to: #358743 (closed)

For the reviewers

  • The method BulkImports::Entity#create_pipeline_trackers! was moved to a service (BulkImports::CreatePipelineTrackersService)
  • BulkImports::Groups::Stage and BulkImports::Projects::Stage specs were written so that the configuration of the pipelines doesn't need to be duplicated in the specs
  • Now, BulkImports::Stage#pipelines returns an array of hashes instead of an array of arrays, and because of that some classes and specs were updated to work with the new method response

Before

BulkImports::Stage#pipeline

[
  [0, BulkImports::Groups::Pipelines::GroupPipeline],
  [1, BulkImports::Groups::Pipelines::GroupAttributesPipeline]
  ...
]

After

BulkImports::Stage#pipeline

[
  { stage: 0, pipeline: BulkImports::Groups::Pipelines::GroupPipeline },
  { stage: 1, pipeline: BulkImports::Groups::Pipelines::GroupAttributesPipeline }
  ...
]

Screenshots or screen recordings

MinMaxVersion

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Testing if the pipelines are being skipped is tricky as we need to import from a source instance running an old GitLab version. One option to test in the local environment is to run a second GDK version and change the GitLab version.

Another option, that is easier, is to update the pipeline configuration as shown in the screen recording above

  1. Feature.enable(:bulk_import).
  2. Create a top-level group
  3. Go to /groups/new#import-group-pane page and enter the instance URL and access token (needs to be api & read_repository scope).
  4. Select the newly created group and click Import.
  5. Wait for Group import to complete and verify the imported group.
  6. Verify if the incompatible bulk_import_trackers have status -2 (skipped) in the DB

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rodrigo Tomonari

Merge request reports