We've seen a couple of issues recently where container repository replication is being marked as failed since the secondary has tags which the primary does not. They do seem to be tags which were removed from the primary, which then should be removed from the secondary. in [one issue](https://gitlab.com/gitlab-com/geo-customers/-/issues/219), manually running [Geo::ContainerRepositorySync.new(FOO).execute](https://gitlab.com/gitlab-org/gitlab/-/blob/e746f62dfe5a97f329553ddcad6f04cc8f32402b/ee/app/services/geo/container_repository_sync.rb) on the secondary did not remove an extra tag, which *was* seen in the output from `tags_to_remove`. There is not currently enough information to isolate the cause of the issue, or why it isn't self healing. This issue should be used for information gathering to help isolate the cause. To check if we might be hitting this issue. Run the following on a secondary rails console ```ruby irb(main):003:0> Geo::ContainerRepositoryRegistry.failed.each do |gcr| Geo::ContainerRepositorySync.new(gcr.container_repository).send(:tags_to_remove) end ``` An empty array indicates this is not the issue ```ruby => [] ``` A non-empty array indicates this may be the issue ```ruby => [{:name=>"latest", :digest=>"sha256:fd6259363372476f7b3d49e11fa6a492aed4bc9c6159afaaef2d0197cd2411f0"}] ``` From here, it would be useful to gather some information about the tags via tools like `skopeo` or `docker manifest inspect`