Enable Batch gitaly delete refs in buffered queue causes gitaly errors
Summary
After enabling feature flags in Batch gitaly delete refs in buffered queue it causes an overflow of fatal: prepare: multiple updates for ref 'refs/pipelines/xxxx' not allowed
errors when trying to delete multiple refs of the same name in a single call. This cause causes a an overflow of sidekiq jobs
Slack conversation for reference
Steps to reproduce
-
Update to 16.3
-
Enable features flags:
- pipeline_delete_gitaly_refs_in_batches
- merge_request_delete_gitaly_refs_in_batches
What is the current bug behavior?
- When the same ref is enqueued to deletion many times the cleanup service
BatchedGitRefUpdates::ProjectCleanupWorker
would fail with the error above. - The service runs
project.repository.delete_refs("refs/heads/feature/dependency-proxy", "refs/heads/feature/dependency-proxy")
. - This call raises exception
Gitlab::Git::CommandError: 13:unable to prepare: state update to "prepare" failed: EOF.
.
What is the expected correct behavior?
- The
BatchedGitRefUpdates::ProjectCleanupWorker
succeeds.
Possible fixes
Proper fix:
Fix bug with batched gitaly ref deletion duplic... (!134323 - merged)
Temporary fix (cleanup duplicate refs). Run in rails console
:
BatchedGitRefUpdates::Deletion.where.not(id: BatchedGitRefUpdates::Deletion.group(:project_id, :ref).select('min(id)')).delete_all
Edited by Kamil Trzciński