Skip to content

Geo ensure files moved to object storage are cleaned up

Toon Claes requested to merge tc-geo-migration-object-storage into master

What does this MR do?

This adds a worker that will clean up the files that are moved to object storage, but were already synced to the local file system.

When at some point the primary (and also the secondary, because they go hand-in-hand) enables object storage for CI artifacts, LFS objects, or Attachments, the files on the primary are migrated to the object storage. But the secondary still has those files on disk. The secondary will not migrate those to object storage -- because it might point to the same store or they are replicated using methods more efficient (e.g. s3cmd sync) --- so the secondary should remove those from disk. We don't want the admin to manually run a task on each secondary, therefore this worker is added.

Todo

  • Test the scheduling of the worker
  • Test the removal itself
  • Update item in registry
  • Do some real-life testing

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

Closes gitlab-org/gitlab-ee#4215

Edited by Toon Claes

Merge request reports