Improved administrative tools for validating integrity of backups
Description
We recently restored a backup only to find a substantial number of repos were restored as empty. There were no errors reported during the backup or restore process. Silent failures like this are obviously undesirable!
Some investigation showed that our situation was similar to this issue: https://gitlab.com/gitlab-org/gitlab-ce/issues/28854
Proposal
Provide some form of administration tooling that can be used to automatically validate a backup. The simplest thing I can envision starts with a list of each repo and it's most recent commit hash which can be used by a script to verify each repo is present and up to date.
Alternately, if the cause of repos going missing is something programmatic which can be interacted with then some ruby tooling to query the state of repo(s) and reset critical data would also be an improvement.