Geo: Avoid rereplicating Blobs that already exist

From #209813 (closed) and &7180 (comment 821509584) and &2153 (comment 837863715):

Problem: Restore a GitLab backup onto a new Geo site. Backfill could be much faster.

Problem: Failover and demote the primary. Backfill could be much faster.

Proposal

• Require the Verification feature
• If a blob registry is brand new (maybe like pending? && verification_pending?), and a blob already exists in the expected location, then just mark it synced
• Make sure that rereplication actually does occur, when verification fails => I did test this locally in !96935 (merged) and it worked
• Add tests in shared examples for verified SSF data types

To do

• Doc update is in !140276 (merged)

changed milestone to %14.8

added Enterprise Edition devopssystems groupgeo workflowready for development labels

set weight to 4

added to epic &2153

marked this issue as related to #209813 (closed)

mentioned in issue #209813 (closed)

changed the description

marked this issue as related to #352533

added customer featureenhancement typefeature labels

changed the description

mentioned in epic &2153

added sectioncore platform label

added [deprecated] Accepting merge requests label

mentioned in issue gitlab-org/quality/triage-reports#6425 (closed)

mentioned in issue gitlab-org/quality/triage-reports#6496 (closed)

changed milestone to %14.9

added missed:14.8 label

mentioned in issue gitlab-org/quality/triage-reports#6580 (closed)

mentioned in issue gitlab-org/quality/triage-reports#6687 (closed)

mentioned in issue gitlab-org/quality/triage-reports#6845 (closed)

changed milestone to %14.10

added missed:14.9 label

mentioned in issue gitlab-org/quality/triage-reports#6911 (closed)

Removing milestone since this is not on the Geo build board (it doesn't have geoactive).

removed milestone %14.10

mentioned in merge request !96935 (merged)

removed [deprecated] Accepting merge requests label

added 1 deleted label

@mkozono

Building on this conversation and this issue, I wanted to confirm something:

• Let's say I have a primary that uses object storage. It's been in service for some time and has lots of data.
• I am planning to set up a Geo secondary for the first time.
• The secondary site will use object storage that already contains the data from the primary.
• Let's say that what I care about the most is job artifacts.

Given the Open nature of this issue:

• Is it true that Geo will attempt to replicate the job artifacts?

We do not presently support verification of files in object storage so the first item in this proposal has not been met.

I just want to make sure that the purpose of this issue is understood. I am making conclusions based on the linked conversation which was some time ago and about local storage rather than object storage.

Please let me know if there's any missing info or a better place/way to raise this question.

---

This is on behalf of a Large customer. GitLab team members with access to Zendesk can learn more in the ticket.

@bcarranza In this case, since the files are already accessible via the secondary site, I would make sure that Allow this secondary site to replicate content on Object Storage is disabled. That way, the secondary site won't operate on the object stored files at all. I think that should suffice, but please LMK if I am misunderstanding.

@mkozono Thanks for the quick reply! That is indeed what this customer will do.