Update Direct Transfer to handle project and group path rules
What does this MR do and why?
With Direct Transfer (aka Bulk Imports aka Gitlab Migration), part of the params
supplied for each Group or Project being imported are a destination_slug/destination_name
, which becomes the project/group's path
, and a destination_namespace
, which is a full_path
to the Namespace the entity should be imported to.
The changes here updates how Direct Transfer handles the path
and name
for each Project or Group imported:
- updating where and how validation for these attributes occurs;
- ensuring
path
andfull_path
attributes now conform to rules introduced for Project paths with !80055 (merged), and behind a soon-to-be-removed feature flag for Group paths (see !111017 (merged)); - handling any duplications that already exist or arise as a consequence of transformation.
Previously regex validation for destination_slug
and destination_namespace
occurred on the Model level when each BulkImport::Entity was created. This validation has been moved to the BulkImports::CreateService
, the UI entry point for Direct Transfer, validating the destination_slug
and destination_namespace
before a BulkImport is created and raising an error if invalid.
Moving this validation out of the Entity model allows the importing of Groups and Projects with legacy path names that are now invalid. These are received in the params
of the request, then transformed within the 'transform' stage of the BulkImports pipeline/s, where they are parameterized, to conform to the existing Gitlab::Regex.oci_repository_path_regex
rules for path
s, and made unique if necessary.
Screenshots
Before:
![](/-/project/278964/uploads/a8a086afa8db9bfdd690665c48ce7216/Screenshot_2023-03-24_at_14.32.53.png)
After:
Validation Steps
With the UI only the Group namespace can be tested at the moment.
- navigate to New Group > Import group and add URL and access token for the source instance
- attempt to import a group giving it an invalid name and observe new message:
With the API
3. To test the destination_slug param: using a personal access token with an api
scope, in a terminal make a curl POST request to the gdk.test
bulk_imports endpoint to initiate a direct transfer from gdk.test
to gdk.test
. Make sure the destination_slug
has invalid characters:
curl --location --request POST 'http://gdk.test:3000/api/v4/bulk_imports' \
--header 'Authorization: Bearer [ACCESS TOKEN]' \
--header 'Content-Type: application/json' \
--data-raw '{
"configuration": {
"url": "http://gdk.test:3000",
"access_token": [ACCESS TOKEN]
},
"entities": [
{
"source_full_path": "source_full_path",
"source_type": "group_entity",
"destination_slug": "----your-invalid*****-destination",
"destination_namespace": "your-destination-namespace"
}
]
}'
- Observe the new message:
{"error":"[destination_slug] must not start or end with a special character and must not contain consecutive special characters. It can only contain alphanumeric characters, periods, underscores, and dashes. E.g. 'destination_namespace' not 'destination/namespace'"}%
- repeat but with an invalid
destination namespace
:
curl --location --request POST 'http://gdk.test:3000/api/v4/bulk_imports' \
--header 'Authorization: Bearer [ACCESS TOKEN]' \
--header 'Content-Type: application/json' \
--data-raw '{
"configuration": {
"url": "http://gdk.test:3000",
"access_token": [ACCESS TOKEN]
},
"entities": [
{
"source_full_path": "source_full_path",
"source_type": "group_entity",
"destination_slug": "----your-invalid*****-destination",
"destination_namespace": "your-destination-namespace"
}
]
}'
- observe the new message:
{"error":"entities[0][destination_namespace] must have a relative path structure with no http protocol chars or leading or trailing forward slashes. Path segments must not start or end with a special character, and must not contain consecutive special characters."}%
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #393631 (closed) and #393492 (closed)