Foreground verification of uploads and LFS objects
What does this MR do?
Reworks the gitlab:uploads:check
rake task and introduces a gitlab:lfs:check
rake task to match it.
The CE and EE implementations have gotten rather out of sync, so I also take the opportunity to rework things so they can be more closely aligned. See the associated EE MR for more details: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4753
Are there points in the code the reviewer needs to double check?
This is just a starting point for verification, not the endpoint. For Geo and GitLab.com, we're also going to have a beautiful frontend with background verification, based on the excellent repository verification work recently demoed by @dbalexandre. This is effectively a stop-gap until that's complete.
CI job artifacts aren't included yet, as we don't store a checksum for them anywhere
Why was this MR needed?
It's important to be able to verify that your user-uploaded files are accurate. They may become corrupted due to bitrot or some other error. Since we store checksums in the database, this is simple to do.
A simple foreground rake task for verification is often sufficient for smaller instances.
Screenshots (if relevant)
Does this MR meet the acceptance criteria?
-
Changelog entry added, if necessary -
Documentation created/updated -
Tests added for this feature/bug - Review
-
Has been reviewed by Backend
-
-
Conform by the merge request performance guides -
Conform by the style guides -
Squashed related commits together -
Internationalization required/considered -
End-to-end tests pass ( package-qa
manual pipeline job)
What are the relevant issue numbers?
Related to gitlab-ee#4862