Geo: Primary checksums are not recalculated after deployment refs are created
From https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5957, we see one repository on our Ops instance that is erroneously labeled as failed verification, when in fact, it is up-to-date.
It looks like the problem occurs because there are some refs that are created after a merge request is merged, and the primary doesn't know to do anything about that:
irb(main):002:0> Geo::ProjectRegistry.find_by(project_id: 150).last_repository_verification_ran_at
=> 2019-01-16 06:53:31 UTC
irb(main):003:0> Project.find(150).repository_state.last_repository_verification_ran_at
=> 2019-01-14 15:26:03 UTC
As you can see above, the primary last ran its verification at 2019-01-14 15:26:03 UTC, which coincides when the last push to the repo happened:
irb(main):005:0> Project.find(150).events.last.created_at
=> Mon, 14 Jan 2019 15:22:28 UTC +00:00
However, if you look in the refs/
directory, you can see new refs created after 15:22:28:
root@gitlab-01-inf-ops.c.gitlab-ops.internal:/var/opt/gitlab/git-data/repositories/@hashed/9a/e2/9ae2bdd7beedc2e766c6b76585530e16925115707dc7a06ab5ee4aa2776b2c7b.git/refs/environments/staging/deployments# ls -al | grep 15:2
-rw-r--r-- 1 git git 41 Jan 14 15:25 1696
-rw-r--r-- 1 git git 41 Jan 14 15:25 1697
-rw-r--r-- 1 git git 41 Jan 14 15:25 1698
-rw-r--r-- 1 git git 41 Jan 14 15:25 1699
-rw-r--r-- 1 git git 41 Jan 14 15:28 1704
-rw-r--r-- 1 git git 41 Jan 14 15:27 1705
-rw-r--r-- 1 git git 41 Jan 14 15:27 1706
-rw-r--r-- 1 git git 41 Jan 14 15:27 1707