Geo: Constantly reverify repositories
Problem
Due to bugs or transient infrastructure problems, it is possible for git repos to change unexpectedly without being marked for verification.
Possible solution
When there is available capacity, we should reverify non-failed repos.
We need to rate limit this work. It should scale with the number of repos, but shouldn't be wasteful, and ideally admins can tweak the limit. Perhaps we could provide a setting like "maximum days since last verification".
Case 1
For the GCP Migration, we manually marked all repos and wikis for reverification. But for ~"Geo DR" we should try to keep all repos verified as recently as possible.
Case 2
Some customers have missing repos and wikis due to https://gitlab.com/gitlab-org/gitlab-ee/issues/6012. If we don't implement reverification, then we need to do a one-off background migration to fix these repos.
Case 3
Some customers may have projects with stale checksums on the primary. A cause of stale checksums is fixed, but customers need to reverify all repos to fix existing stale checksums. https://gitlab.com/gitlab-org/gitlab-ee/issues/7213#note_99514064
-
If closing this issue without resolving it, open a separate issue to do a one-off background migration to fix missing repos/wikis on the secondary. To handle Case 2 -
If closing this issue without resolving it, open a separate issue to do a one-off background migration to reverify repos/wikis on the primary. To handle Case 3