Failed changes when primary is down still increment generation and cause repo to become read-only

When the primary Gitaly node has gone down but is still considered primary by Praefect, mutator requests received during this period will increment the generation column in storage_repositories. As a result, when a new primary is elected it will be treated as 'behind' the old primary despite no changes having gone through on the old one.

Steps to reproduce:

• Gitaly-1 is current primary
• Stop Gitaly-1 with gitlab-ctl stop gitaly
• Immediately afterwards try to push to a repository
  • This push will fail because the primary is down
• After three failed health checks Praefect elects Gitaly-2 as primary
• Attempt to push again, an error that the repo is read-only is received
• The actual repo is in sync across all Gitaly nodes, this is purely due to Praefect eagerly incrementing the generation count

The repo will become accessible again once the old primary comes back online and can replicate to the other nodes, though there are no actual changes to replicate.

Praefect DB has incremented the generation column:

 virtual_storage |                                   relative_path                                    | storage  | generation
-----------------+------------------------------------------------------------------------------------+----------+------------
 default         | @hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git | gitaly-2 |          4
 default         | @hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git | gitaly-3 |          4
 default         | @hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git | gitaly-1 |          5

But checking the repos shows that they are actually in sync:

gitaly-1 $ /opt/gitlab/embedded/git for-each-ref | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -

gitaly-2 $ /opt/gitlab/embedded/git for-each-ref | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -

gitaly-3 $ /opt/gitlab/embedded/git for-each-ref | sha256sum
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  -