Direct Transfer - custom settings per top level group
To prevent misuse, both direct transfer and File-based Import via remote storage and AWS S3 set maximum download sizes of 5GB for Direct Transfer and 10GB for File-based Import.
Note that these limits are already controlled via platform settings bulk_import_max_download_file_size
(Direct Transfer) and max_import_remote_file_size
(AWS S3 API), but on .com
, if we change these limits, the change applies to all new migrations, and we can't control that per each customer.
Also, this feature won't be necessary for GitLab Dedicated and SM in most cases since they can change the application settings globally rather than for each top-level group.
Problem
There are valid situations where PS and Customer Support must import repositories larger than the allowed limit. In such cases, it would be beneficial for them to bypass these limits.
Proposed solution
Option 1: Allow custom limits per customer. The limit could be adjusted like we plan to do for the Placeholder user limit, which is to add a new setting to /admin/groups/<group>/edit
form.
Option 2: Allow these limits to be bypassed when an admin starts the migration
Option 3 (selected): Allow these limits to be bypassed when an admin starts the migration, and an ops feature flag is enabled
Note that, as mentioned in #519759 (comment 2371352760), this solution merely removes the file size validation; both Direct Transfer and File-based Import will still be constrained by the available Sidekiq disk size for processing the files.
Docs
Update docs and link in the Importers FAQ. Announce in PS and Support channels.