Something went wrong while fetching comments. Please try again.
Geo: Document 14.6 unnecessary resyncing due to new verification feature
Problem
From #334550 (comment 887807869):
In 14.6 we added the Verification feature to LFS objects. I wonder if it's like this:
- Upgrade to 14.6
- The primary site starts checksumming LFS objects
- The secondary site starts checksumming LFS objects and comparing them to primary site checksums
- But most LFS objects are not checksummed yet on the primary
- Therefore many or most (but probably not all) LFS objects start "failing" verification on the secondary site
- A failed verification schedules a resync, therefore many or most LFS objects get resynced
This affects at least Uploads also.
Proposal
- Determine what data types are affected => uploads, LFS objects, Pages deployments, and external merge request diffs, see https://about.gitlab.com/releases/2021/12/22/gitlab-14-6-released/#geo-verifies-four-additional-data-types
- Determine what releases are affected => 14.6, 14.7
-
Determine a workaround
- Deploy GitLab 14.5
- Store files locally, not in object storage, so that verification will occur
- Create at least a few of each data type: uploads, LFS objects, Pages deployments, and external merge request diffs
- Disable auto migrate if enabled
-
Upgrade a node with the
gitlab-rails
service to 14.6 - Run pre-deployment migrations only
- Try this potential workaround: Run update SQL queries which mark uploads, LFS objects, Pages deployments, and external merge request diffs as "verified".
- Finish upgrading to 14.6
- List the registry tables of affected data types
- Look for failed verifications or resyncing. There should be none if the workaround was effective.
- Run queries to mark all the records of affected data types as "pending verification". Mention that this is optional since re-verification will automatically cause verification of everything, eventually.
-
Document the problem and workaround in Geo Version-specific Update instructions doc
- If multiple sections are needed in Geo Version-specific Update instructions doc, then put the problem and workaround in Geo Troubleshooting and link to that.