Default enable invalid metadata deletion in Praefect
Praefect has a metadata verifier that checks whether the replicas recorded in the database actually exist. For now, it has only logged and gathered metrics on invalid metadata records by default. The verifier also supports deleting invalid metadata which clean up the stale records and allows Praefect to realize the replicas are missing and trigger replication. This functionality has been disabled in the past as renames racing with the verificaton could cause data loss. Since 15.3, Praefect has been using Praefect generate replica paths and not performing any more renames on disk. Given this, there are no more races that we know of, so we can default enable the invalid metadata deletion.
Changelog: changed
Closes #4429 (closed)
Documentation gitlab!110661 (merged)
Praefect generated paths were default enabled in !4809 (merged). Enabling this functionality stopped renames from being scheduled. !4941 (merged) removed the feature flag so everyone has the pre-requisite set.
Merge request reports
Activity
added Category:Gitaly groupgitaly sectioncore platform + 1 deleted label
assigned to @samihiltunen
- A deleted user
changed milestone to %15.9
- A deleted user
added devopssystems typemaintenance labels
Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category and by a maintainer in all other categories.
To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.
Category Reviewer Maintainer ~ Patrick Steinhardt (
@pks-t
) (UTC+1, 1 hour behind@samihiltunen
)Quang-Minh Nguyen (
@qmnguyen0711
) (UTC+7, 5 hours ahead of@samihiltunen
)If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangerremoved devopssystems typemaintenance labels
- Resolved by 🤖 GitLab Bot 🤖
Proper labels assigned to this merge request. Please ignore me.
@samihiltunen - please see the following guidance and update this merge request.1 Error Please add typebug typefeature, or typemaintenance label to this merge request. Edited by 🤖 GitLab Bot 🤖
added featureaddition typefeature labels
requested review from @toon and @jcaigitlab
- Resolved by Sami Hiltunen
@samihiltunen change looks good to me, but could you add some context in the description explaining that we needed invalid metadata detection in the first place due to a race that led to such invalid metadata being written?
And maybe a link back to the commit that changed renames. That would be helpful historical context if ever we had to go back to this MR. thanks!
removed review request for @jcaigitlab
@samihiltunen Ah cool, nice to see this happen.
- A deleted user
added devopssystems typemaintenance labels and removed typefeature label
removed featureaddition label
mentioned in commit 10880feb
added workflowstaging-canary label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
mentioned in issue #5089 (moved)
mentioned in issue #5506 (moved)