Skip to content

Foreground verification of uploads and LFS objects

Nick Thomas requested to merge ee-4862-verify-file-checksums into master

What does this MR do?

Reworks the gitlab:uploads:check rake task and introduces a gitlab:lfs:check rake task to match it.

The CE and EE implementations have gotten rather out of sync, so I also take the opportunity to rework things so they can be more closely aligned. See the associated EE MR for more details: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4753

Are there points in the code the reviewer needs to double check?

This is just a starting point for verification, not the endpoint. For Geo and GitLab.com, we're also going to have a beautiful frontend with background verification, based on the excellent repository verification work recently demoed by @dbalexandre. This is effectively a stop-gap until that's complete.

CI job artifacts aren't included yet, as we don't store a checksum for them anywhere 😅 . I plan to address this in a follow-up MR which will make use of the framework provided in this one.

Why was this MR needed?

It's important to be able to verify that your user-uploaded files are accurate. They may become corrupted due to bitrot or some other error. Since we store checksums in the database, this is simple to do.

A simple foreground rake task for verification is often sufficient for smaller instances.

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

Related to gitlab-ee#4862

Edited by Nick Thomas

Merge request reports