Skip to content

After 18.4 all gitaly.CommitService/CommitLanguages requests fail transactions with no-votes causing constant replication

We're running a self-hosted GitLab Premium in K8s. Immediately after we updated to 18.4 two days ago we began seeing a huge increase in no-votes and outdated tx result values reported through the gitaly_praefect_tx_replications_total metric and "transactional node states" log message in Praefect. We have been able to narrow down these issues to be occurring only with gitaly.CommitService/CommitLanguages and gitaly.RepositoryService/FetchRemote. With gitaly.RepositoryService/FetchRemote it seems like we were having them even before the update, and they're rare, so we didn't see any impact from them. With gitaly.CommitService/CommitLanguages, however, they occur on literally every commit to any repo (each commit results in a gitaly.CommitService/CommitLanguages call from Gitaly which then fails with no-votes and causes the replicas to be resynced).

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information