Skip to content

Serialize delete refs for Merge Requests to avoid Gitaly deadlock

Kamil Trzciński requested to merge merge-request-cleanup-ref-async into master

What does this MR do and why?

Serialize delete refs for Merge Requests to avoid Gitaly deadlock.

This is a fix similar to !123976 (merged). It changes both to use the same serialization mechanism to reduce the likelity of Gitaly deadlock as described in gitaly#5368.

This MR does:

  • Build on top of work in !123976 (merged).
  • Serializes all references managed by merge-requests: refs/merge-requests/IID/head, refs/merge-requests/IID/merge, refs/merge-requests/IID/train.
  • They are scheduled and run async, synchronized with deleted being done by CI Pipelines.
  • This should cover 99.99% of cases when deletes does happen in code.

Remarks:

  • Those deletes still happen one-by-one, so they do generate extra pressure on Gitaly.
  • This does show case all places where deletes do happen and generalises their handling making it easier to implement: gitaly#5369 (closed).
  • This has its own feature flag: merge_request_cleanup_ref_worker_async. To avoid deadlock the merge_request_cleanup_ref_worker_async and pipeline_cleanup_ref_worker_async should be enabled.

How to set up and validate locally

  1. In rails console enable the experiment fully 
    Feature.enable(: merge_request_cleanup_ref_worker_async)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Grzegorz Bizon

Merge request reports