Geo: Container repos failure: "Checksum does not match the primary checksum"

Problem

Like #427493 (comment 1777125097) but for container repos.

In https://gitlab.com/gitlab-com/geo-customers/-/issues/194:

They are seeing Verification failed with: Checksum does not match the primary checksum for container repositories (70), we tried reverifying and resyncing but it just goes back to failing:

Primary:

irb(main):454:1* failed = [904,
irb(main):455:1*  923,
irb(main):456:1*  945,
...
irb(main):535:0> failed.each { |f| ContainerRepository.find(f).replicator.verify }
Secondary:

irb(main):033:0> Geo::ContainerRepositoryRegistry.failed.where("last_sync_failure like 'Verification failed with: Checksum does not match%'").update_all(retry_at: 1.hour.ago)
=> 70

After a couple of minutes, the UI showed 0 queued and 1 failed repository.

After ~1 hour we have 5 failed repositories.
After ~2 hours we have 15 failed repositories.
After ~3 hours we have 22 failed repositories.
After ~4 hours we have 26 failed repositories.