Serialize delete refs for Merge Requests to avoid Gitaly deadlock
What does this MR do and why?
Serialize delete refs for Merge Requests to avoid Gitaly deadlock.
This is a fix similar to !123976 (merged). It changes both to use the same serialization mechanism to reduce the likelity of Gitaly deadlock as described in gitaly#5368 (closed).
This MR does:
- Build on top of work in !123976 (merged).
- Serializes all references managed by merge-requests:
refs/merge-requests/IID/head
,refs/merge-requests/IID/merge
,refs/merge-requests/IID/train
. - They are scheduled and run async, synchronized with deleted being done by
CI Pipelines
. - This should cover 99.99% of cases when deletes does happen in code.
Remarks:
- Those deletes still happen one-by-one, so they do generate extra pressure on Gitaly.
- This does show case all places where deletes do happen and generalises their handling making it easier to implement: gitaly#5369 (closed).
- This has its own feature flag:
merge_request_cleanup_ref_worker_async
. To avoid deadlock themerge_request_cleanup_ref_worker_async
andpipeline_cleanup_ref_worker_async
should be enabled.
How to set up and validate locally
- In rails console enable the experiment fully
Feature.enable(: merge_request_cleanup_ref_worker_async)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Merge request reports
Activity
assigned to @ayufan
added 1 commit
- 6f981800 - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
changed milestone to %16.2
added bugperformance grouptenant scale typebug labels
added devopsdata stores sectioncore platform labels
requested review from @grzesiek
- A deleted user
added backend feature flag labels
1 Message CHANGELOG missing: If you want to create a changelog entry for GitLab FOSS, add the
Changelog
trailer to the commit message you want to add to the changelog.If you want to create a changelog entry for GitLab EE, also add the
EE: true
trailer to your commit message.If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Piotr Skorupa (
@pskorupa
) (UTC+2, same timezone as@ayufan
)Tetiana Chupryna (
@brytannia
) (UTC+2, same timezone as@ayufan
)~"Verify" Reviewer review is optional for ~"Verify" Drew Cimino (
@drew
) (UTC+0, 2 hours behind@ayufan
)To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
Sidekiq queue changes
This merge request contains changes to Sidekiq queues. Please follow the documentation on changing a queue's urgency.
These queues were added:
merge_requests_cleanup_ref
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangeradded 1 commit
- 245eef86 - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@f79a3756
added 1 commit
- 0d513abb - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@5b08141b
added 1 commit
- 65e51ced - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@bc6cce32
Allure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for 41d49e0dexpand test summary
+-----------------------------------------------------------------------+ | suites summary | +------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------+--------+--------+---------+-------+-------+--------+ | Plan | 4 | 0 | 0 | 4 | 4 | ❗ | | Create | 8 | 0 | 1 | 5 | 9 | ❗ | | Monitor | 4 | 0 | 0 | 4 | 4 | ❗ | | Framework sanity | 0 | 0 | 1 | 0 | 1 | ➖ | | Manage | 1 | 0 | 0 | 0 | 1 | ✅ | | Data Stores | 2 | 0 | 0 | 0 | 2 | ✅ | | Govern | 2 | 0 | 0 | 0 | 2 | ✅ | +------------------+--------+--------+---------+-------+-------+--------+ | Total | 21 | 0 | 2 | 13 | 23 | ❗ | +------------------+--------+--------+---------+-------+-------+--------+
e2e-package-and-test:
test report for 41d49e0dexpand test summary
+------------------------------------------------------------+ | suites summary | +-------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------+--------+--------+---------+-------+-------+--------+ +-------+--------+--------+---------+-------+-------+--------+ | Total | 0 | 0 | 0 | 0 | 0 | ➖ | +-------+--------+--------+---------+-------+-------+--------+
added 1 commit
- 02c121e0 - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@8bfa8c27
marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed
@grzesiek
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
enabled an automatic merge when the pipeline for 8bfa8c27 succeeds
added pipeline:mr-approved label
- Resolved by Grzegorz Bizon
@grzesiek I will cancel auto merge for now. I need to do more manual tests, and ask you to merge this on Monday/Tuesday.
Edited by Kamil Trzciński
removed review request for @grzesiek
added 1 commit
- 30e813e9 - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@97ad7214
added 1 commit
- c6dd2929 - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
requested review from @DylanGriffith and @grzesiek
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@6038dda5
added 1 commit
- 41d49e0d - Serialize delete refs for Merge Requests to avoid Gitaly deadlock
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@867f2832
enabled an automatic merge when the pipeline for f94a319e succeeds
mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@f94a319e
mentioned in commit 92119f3a
added workflowstaging-canary label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
mentioned in merge request !125126 (merged)
mentioned in merge request !126086 (closed)
mentioned in merge request !125333 (merged)
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
mentioned in issue gitlab-org/quality/triage-reports#15297 (closed)
mentioned in issue gitlab-org/quality/triage-reports#15721 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16026 (closed)
mentioned in issue gitlab-org/quality/triage-reports#16488 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17020 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17468 (closed)
mentioned in issue gitlab-org/quality/triage-reports#17927 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18460 (closed)
mentioned in issue gitlab-org/quality/triage-reports#18939 (closed)
mentioned in issue gitlab-org/quality/triage-reports#19393 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20622 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20986 (closed)
mentioned in issue gitlab-org/quality/triage-reports#21551 (closed)
mentioned in issue gitlab-org/quality/triage-reports#22011