Implement `MigrateArtifactsCronWorker`
Problem
When it comes to how artifacts are uploaded to object storage, there are several options. One of the latest ones is Direct Upload. GitLab Workhorse attempts to upload artifact files to object storage and if it failed, it retries until the request succeeded.
On the other hands, in Background Upload or Job trace, if the uploading process failed, the original files remain in local file storage. This is bugging us sometimes like https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6499 as infrastructure engineers have to migrate those files manually by executing rake gitlab:artifacts:migrate
.
Proposal
We implement a cron-type Sidekiq worker to periodically (maybe 0 1 * * *
- nightly job) migrate those leftovers.