Geo: Document 14.6 unnecessary resyncing due to new verification feature
Problem
From #334550 (comment 887807869):
In 14.6 we added the Verification feature to LFS objects. I wonder if it's like this:
- Upgrade to 14.6
- The primary site starts checksumming LFS objects
- The secondary site starts checksumming LFS objects and comparing them to primary site checksums
- But most LFS objects are not checksummed yet on the primary
- Therefore many or most (but probably not all) LFS objects start "failing" verification on the secondary site
- A failed verification schedules a resync, therefore many or most LFS objects get resynced
This affects at least Uploads also.
Proposal
-
Determine what data types are affected => uploads, LFS objects, Pages deployments, and external merge request diffs, see https://about.gitlab.com/releases/2021/12/22/gitlab-14-6-released/#geo-verifies-four-additional-data-types -
Determine what releases are affected => 14.6, 14.7 -
Determine a workaround -
Deploy GitLab 14.5 -
Store files locally, not in object storage, so that verification will occur -
Create at least a few of each data type: uploads, LFS objects, Pages deployments, and external merge request diffs -
Disable auto migrate if enabled -
Upgrade a node with the gitlab-rails
service to 14.6 -
Run pre-deployment migrations only -
Try this potential workaround: Run update SQL queries which mark uploads, LFS objects, Pages deployments, and external merge request diffs as "verified". -
Finish upgrading to 14.6 -
List the registry tables of affected data types -
Look for failed verifications or resyncing. There should be none if the workaround was effective. -
Run queries to mark all the records of affected data types as "pending verification". Mention that this is optional since re-verification will automatically cause verification of everything, eventually.
-
-
Document the problem and workaround in Geo Version-specific Update instructions doc -
If multiple sections are needed in Geo Version-specific Update instructions doc, then put the problem and workaround in Geo Troubleshooting and link to that.
-
Edited by Michael Kozono