Skip to content

Direct Transfer - custom settings per top level group

To prevent misuse, both direct transfer and File-based Import via remote storage and AWS S3 set maximum download sizes of 5GB for Direct Transfer and 10GB for File-based Import.

Note that these limits are already controlled via platform settings bulk_import_max_download_file_size (Direct Transfer) and max_import_remote_file_size (AWS S3 API), but on .com, if we change these limits, the change applies to all new migrations, and we can't control that per each customer.

Also, this feature won't be necessary for GitLab Dedicated and SM in most cases since they can change the application settings globally rather than for each top-level group.

Problem

There are valid situations where PS and Customer Support must import repositories larger than the allowed limit. In such cases, it would be beneficial for them to bypass these limits.

Proposed solution

Option 1: Allow custom limits per customer. The limit could be adjusted like we plan to do for the Placeholder user limit, which is to add a new setting to /admin/groups/<group>/edit form.

Option 2: Allow these limits to be bypassed when an admin starts the migration

Option 3 (selected): Allow these limits to be bypassed when an admin starts the migration, and an ops feature flag is enabled

Note that, as mentioned in #519759 (comment 2371352760), this solution merely removes the file size validation; both Direct Transfer and File-based Import will still be constrained by the available Sidekiq disk size for processing the files.

Docs

Update docs and link in the Importers FAQ. Announce in PS and Support channels.

Edited by Rodrigo Tomonari