Clarify the handling of non-transactional mutator RPCs

@zj-gitlab @pks-t in the context of #3605 (closed), I went through the non-transactional mutator RPCs we have that trigger update replication. Any such RPC triggers replication and causes the secondaries to be considered outdated until the replication has taken place. The replication syncs the attributes file and performs a git fetch. Some of these RPCs cause secondaries to be considered outdated unnecessarily while some of them would not seemingly get the changes replicated properly to secondaries.

  1. Optimization RPCs that should not cause secondaries to be outdated and do not get applied to secondaries as it is:

    • /gitaly.RepositoryService/WriteCommitGraph
    • /gitaly.RepositoryService/OptimizeRepository
    • /gitaly.RepositoryService/MidxRepack

  2. /gitaly.RepositoryService/Fsck, this should not even be a mutator? The expected result of this RPC in the context of Praefect having multiple replicas is not clear, given we're only reporting the status from one of the replicas.

  3. Depending on the config set, the repository may handle operations differently. It doesn't appear the config is propagated through the replication:

    • /gitaly.RepositoryService/DeleteConfig
    • /gitaly.RepositoryService/SetConfig

  4. /gitaly.RepositoryService/RestoreCustomHooks, unpacks a tar to the disk. Only the primary would get the hooks, they don't seem to get replicated to the secondaries.

  5. The pool operations are currently applied only to the primary which leaves the status of these RPCs somewhat unclear. Looks like the replication would target a pool repo on the secondary.

    • /gitaly.ObjectPoolService/CreateObjectPool, this should get replicated via the usual handling of update jobs as it only creates a repository.
    • /gitaly.ObjectPoolService/DeleteObjectPool, this is not handled specially, thus would trigger usual replication. As such, pools would be left on disk on the secondaries. Also, database records would be left for these repositories as the RPC is not handled as a delete by Praefect.
    • /gitaly.ObjectPoolService/UnlinkRepositoryFromObjectPool, unlinks the repo from the pool on the primary, secondaries would just fetch.
    • /gitaly.ObjectPoolService/LinkRepositoryToObjectPool, links the repo to a pool on the primary, secondaries would just fetch.
    • /gitaly.ObjectPoolService/DisconnectGitAlternates, disconnects the repo from the pool on the primary, secondaries would just fetch.
    • /gitaly.ObjectPoolService/ReduplicateRepository, seems to be just a repack and thus would not get applied to secondaries on replication.

We should handle these in some different manner to ensure they get properly replicated and don't unnecessarily trigger replication and make the secondaries outdated.

Edited by Sami Hiltunen