Geo: repository verification on the *primary* sometimes gives the wrong checksum

Summary

Sometimes, a Geo primary calculates a wrong (or perhaps outdated?) checksum for a repository. Observed with ~60 projects during the GCP Migration. So it's rare, perhaps a race or related to the data source?

Steps to reproduce

???

Example Project

See https://gitlab.com/gitlab-com/migration/issues/546#note_93833953

What is the current bug behavior?

project.repository_state.repository_verification_checksum == project.repository.checksum == false

What is the expected correct behavior?

project.repository_state.repository_verification_checksum == project.repository.checksum == true

Output of checks

This bug happens on GitLab.com

Possible fixes

Unsure

/cc @stanhu @dbalexandre @akaemmerle