Add local project uploads cleanup task
What does this MR do?
Adds a rake task that can list, or delete move to lost-and-found, orphaned local project upload files.
Are there points in the code the reviewer needs to double check?
Why was this MR needed?
It is necessary for https://gitlab.com/gitlab-org/gitlab-ce/issues/49630 and https://gitlab.com/gitlab-org/gitlab-ee/issues/6012
Screenshots (if relevant)
Example output:
$ sudo gitlab-rake gitlab:cleanup:project_uploads
I, [2018-07-27T12:08:28.671559 #89817] INFO -- : Looking for orphaned project uploads to clean up. Dry run...
E, [2018-07-27T12:08:28.689869 #89817] ERROR -- : Skipping... Unable to parse project upload path: "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out"
I, [2018-07-27T12:08:28.754259 #89817] INFO -- : Found correct path! /opt/gitlab/embedded/service/gitlab-rails/public/uploads/qux/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt
I, [2018-07-27T12:08:28.755624 #89817] INFO -- : Can be moved to parent: /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt
I, [2018-07-27T12:08:28.760257 #89817] INFO -- : Can be moved to lost and found: /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png
I, [2018-07-27T12:08:28.764470 #89817] INFO -- : To cleanup these files run this command with DRY_RUN=false
$ sudo gitlab-rake gitlab:cleanup:project_uploads DRY_RUN=false
I, [2018-07-27T12:09:24.944414 #89936] INFO -- : Looking for orphaned project uploads to clean up...
E, [2018-07-27T12:09:24.962321 #89936] ERROR -- : Skipping... Unable to parse project upload path: "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/test.out"
I, [2018-07-27T12:08:28.754259 #89817] INFO -- : Found correct path! /opt/gitlab/embedded/service/gitlab-rails/public/uploads/qux/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt
I, [2018-07-27T12:08:28.755624 #89817] INFO -- : Moved to parent: /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/89a0f7b0b97008a4a18cedccfdcd93fb/foo.txt
I, [2018-07-27T12:08:28.760257 #89817] INFO -- : Moved to lost and found: /opt/gitlab/embedded/service/gitlab-rails/public/uploads/foo/bar/1dd6f0f7eefd2acc4c2233f89a0f7b0b/image.png
Does this MR meet the acceptance criteria?
-
Changelog entry added, if necessary -
Documentation created/updated -
Tests added for this feature/bug - Conforms to the code review guidelines
-
Has been reviewed by a Backend maintainer
-
-
Conforms to the merge request performance guidelines -
Conforms to the style guides -
Conforms to the database guides -
If you have multiple commits, please combine them into a few logically organized commits by squashing them -
End-to-end tests pass ( package-and-qa
manual pipeline job)
What are the relevant issue numbers?
https://gitlab.com/gitlab-org/gitlab-ce/issues/49630 and https://gitlab.com/gitlab-org/gitlab-ee/issues/6012
Edited by Michael Kozono