Geo ensure files moved to object storage are cleaned up
What does this MR do?
This adds a worker that will clean up the files that are moved to object storage, but were already synced to the local file system.
When at some point the primary (and also the secondary, because they go hand-in-hand) enables object storage for CI artifacts, LFS objects, or Attachments, the files on the primary are migrated to the object storage. But the secondary still has those files on disk. The secondary will not migrate those to object storage -- because it might point to the same store or they are replicated using methods more efficient (e.g. s3cmd sync
) --- so the secondary should remove those from disk. We don't want the admin to manually run a task on each secondary, therefore this worker is added.
Todo
-
Test the scheduling of the worker -
Test the removal itself -
Update item in registry -
Do some real-life testing
Does this MR meet the acceptance criteria?
-
Changelog entry added, if necessary -
Documentation created/updated -
API support added -
Tests added for this feature/bug - Review
-
Has been reviewed by UX -
Has been reviewed by Frontend -
Has been reviewed by Backend -
Has been reviewed by Database
-
-
Conform by the merge request performance guides -
Conform by the style guides -
Squashed related commits together -
Internationalization required/considered -
If paid feature, have we considered GitLab.com plan and how it works for groups and is there a design for promoting it to users who aren't on the correct plan -
End-to-end tests pass ( package-qa
manual pipeline job)
What are the relevant issue numbers?
Closes gitlab-org/gitlab-ee#4215
Edited by Toon Claes