Geo Container Registry Repository Replication Sync and Geo Container Registry Repository Sync Check Disagree

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When syncing container registry repositories, Geo will:

  • take the list of tags on the primary and secondary
  • sync any tags only on the primary to the secondary

The logic that confirms that a repository is synced

  • creates an array of tuples of tags and the manifest SHA they reference
  • casts this list into a string
  • creates a SHA of that string
  • compares the SHA from both the primary and the secondary side

In this way, the check logic is far stricter than the sync logic and there are circumstances where a sync can complete without error, but the check logic will indicate, correctly, that the repositories are not in sync. Namely, these circumstances are:

  • a tag exists on the secondary and not the primary
  • a tag exists on both the secondary and the primary, but the tag references a different manifest

The secondary having extra tags is not ideal, but not critically bad; however, the secondary having the same tag pointing to a different manifest is rather serious and should be avoided. This discrepancy is difficult to detect both in the UI and via container image CLI tools, unless you're actively looking for this.

Steps to reproduce

Extra Tag

  1. Brake container registry replication
  2. Push a tag to the secondary with a unique name that does not exist on the primary
  3. Restore container registry replication
  4. Resync the container registry repository
  5. The repositories should eventually report as being out of sync

Tag Point to a Different Manifest

  1. Brake container registry replication
  2. Repush a tag to the secondary with a name that exists on the primary, but targeting a different image
  3. Restore container registry replication
  4. Resync the container registry repository
  5. The repositories should eventually report as being out of sync

Example Project

What is the current bug behavior?

The Geo sync logic does not account for tag mutability, i.e. the same tag can point to different manifests over the course of its lifetime. Nor does it try to remove tags on the secondary that do not exists on the primary.

What is the expected correct behavior?

The sync logic should remove tags on the secondary that are not on the primary.

The logic should also compare tag and SHA tuples, like the check logic, and ensure that any tag that differs on the secondary is repushed from the primary, the same as if it were not present.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by 🤖 GitLab Bot 🤖