Skip to content

Update Direct Transfer to handle project and group path rules

Carla Drago requested to merge 393631-update-dest-namespace-regex into master

What does this MR do and why?

With Direct Transfer (aka Bulk Imports aka Gitlab Migration), part of the params supplied for each Group or Project being imported are a destination_slug/destination_name, which becomes the project/group's path, and a destination_namespace, which is a full_path to the Namespace the entity should be imported to.

The changes here updates how Direct Transfer handles the path and name for each Project or Group imported:

  • updating where and how validation for these attributes occurs;
  • ensuring path and full_path attributes now conform to rules introduced for Project paths with !80055 (merged), and behind a soon-to-be-removed feature flag for Group paths (see !111017 (merged));
  • handling any duplications that already exist or arise as a consequence of transformation.

Previously regex validation for destination_slug and destination_namespace occurred on the Model level when each BulkImport::Entity was created. This validation has been moved to the BulkImports::CreateService, the UI entry point for Direct Transfer, validating the destination_slug and destination_namespace before a BulkImport is created and raising an error if invalid.

Moving this validation out of the Entity model allows the importing of Groups and Projects with legacy path names that are now invalid. These are received in the params of the request, then transformed within the 'transform' stage of the BulkImports pipeline/s, where they are parameterized, to conform to the existing Gitlab::Regex.oci_repository_path_regex rules for paths, and made unique if necessary.

Screenshots

Before:

After:

Screenshot_2023-03-28_at_17.18.06

Validation Steps

With the UI only the Group namespace can be tested at the moment.

  1. navigate to New Group > Import group and add URL and access token for the source instance
  2. attempt to import a group giving it an invalid name and observe new message: Screenshot_2023-03-28_at_17.18.06

With the API 3. To test the destination_slug param: using a personal access token with an api scope, in a terminal make a curl POST request to the gdk.test bulk_imports endpoint to initiate a direct transfer from gdk.test to gdk.test. Make sure the destination_slug has invalid characters:

curl --location --request POST 'http://gdk.test:3000/api/v4/bulk_imports' \
--header 'Authorization: Bearer [ACCESS TOKEN]' \
--header 'Content-Type: application/json' \
--data-raw '{
    "configuration": {
        "url": "http://gdk.test:3000",
        "access_token": [ACCESS TOKEN]
    },
    "entities": [
        {
            "source_full_path": "source_full_path",
            "source_type": "group_entity",
            "destination_slug": "----your-invalid*****-destination",
            "destination_namespace": "your-destination-namespace"
        }
    ]
}'
  1. Observe the new message:
{"error":"[destination_slug] must not start or end with a special character and must not contain consecutive special characters. It can only contain alphanumeric characters, periods, underscores, and dashes. E.g. 'destination_namespace' not 'destination/namespace'"}%
  1. repeat but with an invalid destination namespace:
curl --location --request POST 'http://gdk.test:3000/api/v4/bulk_imports' \
--header 'Authorization: Bearer [ACCESS TOKEN]' \
--header 'Content-Type: application/json' \
--data-raw '{
    "configuration": {
        "url": "http://gdk.test:3000",
        "access_token": [ACCESS TOKEN]
    },
    "entities": [
        {
            "source_full_path": "source_full_path",
            "source_type": "group_entity",
            "destination_slug": "----your-invalid*****-destination",
            "destination_namespace": "your-destination-namespace"
        }
    ]
}'
  1. observe the new message:
{"error":"entities[0][destination_namespace] must have a relative path structure with no http protocol chars or leading or trailing forward slashes. Path segments must not start or end with a special character, and must not contain consecutive special characters."}%

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #393631 (closed) and #393492 (closed)

Edited by Carla Drago

Merge request reports