Geo: how to handle repositories/wikis that fail verification
How should we handle verification failures on both the primary and secondary? From discussion on https://gitlab.com/gitlab-org/gitlab-ee/issues/5420
In terms of re-verifying failures...for repositories that never change anymore, with successfully calculated checksums on the primary, but failed checksums on the secondary. What's the distinguishing bit of data that keeps us from querying the same 1000 records over and over?
And what do we want to happen? Just keep re-verifying these repos add infinitum? Not re-verify and expect an admin to intervene? Every few days reset the errors and let it run through the normal verification process again (still add infinitum but not constantly re-verifying)?
Maybe we should have a max retry value? For repositories that exceed this value, we should not re-verify and expect an admin to intervene.