Skip to content
Snippets Groups Projects

Serialize delete refs for Merge Requests to avoid Gitaly deadlock

Merged Kamil Trzciński requested to merge merge-request-cleanup-ref-async into master

What does this MR do and why?

Serialize delete refs for Merge Requests to avoid Gitaly deadlock.

This is a fix similar to !123976 (merged). It changes both to use the same serialization mechanism to reduce the likelity of Gitaly deadlock as described in gitaly#5368 (closed).

This MR does:

  • Build on top of work in !123976 (merged).
  • Serializes all references managed by merge-requests: refs/merge-requests/IID/head, refs/merge-requests/IID/merge, refs/merge-requests/IID/train.
  • They are scheduled and run async, synchronized with deleted being done by CI Pipelines.
  • This should cover 99.99% of cases when deletes does happen in code.

Remarks:

  • Those deletes still happen one-by-one, so they do generate extra pressure on Gitaly.
  • This does show case all places where deletes do happen and generalises their handling making it easier to implement: gitaly#5369 (closed).
  • This has its own feature flag: merge_request_cleanup_ref_worker_async. To avoid deadlock the merge_request_cleanup_ref_worker_async and pipeline_cleanup_ref_worker_async should be enabled.

How to set up and validate locally

  1. In rails console enable the experiment fully
    Feature.enable(: merge_request_cleanup_ref_worker_async)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Grzegorz Bizon

Merge request reports

Merged results pipeline #918473576 passed with warnings

Pipeline: GitLab

#918479958

    Pipeline: Ruby 3 forced pipeline

    #918479961

      Pipeline: E2E Omnibus GitLab EE

      #918490110

        +2

        Merged results pipeline passed with warnings for f94a319e

        Test coverage 82.48% (15.72%) from 2 jobs
        Approved by

        Merged by Grzegorz BizonGrzegorz Bizon 1 year ago (Jul 2, 2023 3:04pm UTC)

        Merge details

        Pipeline #918678654 passed with warnings

        Pipeline: TRIGGER_CACHE_UPDATE_PIPELINE

        #918679981

          Pipeline: GitLab

          #918679770

            Pipeline: E2E GDK

            #918683827

              +1

              Pipeline passed with warnings for 92119f3a on master

              Test coverage 74.53% (15.72%) from 2 jobs
              10 environments impacted.

              Activity

              Filter activity
              • Approvals
              • Assignees & reviewers
              • Comments (from bots)
              • Comments (from users)
              • Commits & branches
              • Edits
              • Labels
              • Lock status
              • Mentions
              • Merge request status
              • Tracking
              Please register or sign in to reply
              Loading