Racy writing of refs in `refs/pipelines` causes frequent locking issues
The way Rails updates references in refs/pipelines/
is causing issues in Gitaly. Seemingly, there frequently are multiple jobs which try to create those references at the same time which then lead to a failure to acquire locks in git-update-ref(1) on Gitaly's side. Given that this is racy behaviour the consequence is that we only observe this error by chance, which causes us to schedule replication jobs in Gitaly Cluster setups.
Elastic shows that over the last 12 hours alone we had 2669 of such events: https://log.gprd.gitlab.net/goto/039528f0-8da5-11ec-a649-b7cbb8e4f62e.