Investigate: Avoid the copy operation during Package Registry uploads
🔥 Problem
Similar to Avoid copying objects from one bucket to anothe... (#285597 - closed), the package registry can receive uploads for large files. The majority of the package registry uploads (all formats) will use a workhorse direct upload. In this mode, the file is put on object storage on a temporary location and when the upload is confirmed by the backend, the file is moved to its final location (using a copy operation).
The problem is that the GitLab instance can be connected to different object storage providers and that copy operation can take more or less time depending on the file size.
🚒 Solution
- Apply the same solution as Avoid copying objects from one bucket to anothe... (#285597 - closed) to completely avoid the
copy
operation. - We will need a new column in the package files table.
-
⚠ Some package files are moved between final destinations in background workers. Those workers will need to play nice with this change. Example: the NuGet background worker extractor. - Make sure that the changes works with bucket prefixes.
- This change needs a feature flag and a close monitoring during the rollout.
🔮 Other considerations
The above change will implement some parts of #358941.
Edited by David Fernandez