geo.staging.gitlab.com: Some snippet repositories failing verification

I assumed any failures would be resolved by #301244 (closed), which is not yet implemented. But I resynced a failing snippet repository, and it did not successfully verify after that. The snippet looks the same in the web UI on the primary and secondary.

https://staging.gitlab.com/-/snippets/1706247

irb(main):001:0> Geo::SnippetRepositoryRegistry.failed.first
=> nil
irb(main):002:0> Geo::SnippetRepositoryRegistry.verification_failed.first
=> #<Geo::SnippetRepositoryRegistry id: 17028, retry_at: nil, last_synced_at: "2021-04-27 17:57:41.267460000 +0000", created_at: "2021-04-27 16:38:21.179499000 +0000", snippet_repository_id: 1706247, state: 2, retry_count: 0, last_sync_failure: nil, force_to_redownload: false, missing_on_primary: false, verification_started_at: "2021-06-12 00:05:13.154482000 +0000", verified_at: "2021-06-12 00:05:15.302098000 +0000", verification_retry_at: "2021-06-12 01:05:59.302057000 +0000", verification_retry_count: 1069, verification_state: 3, checksum_mismatch: true, verification_checksum: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_checksum_mismatched: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_failure: "Checksum does not match the primary checksum {:che...">
irb(main):003:0> srr = _
=> #<Geo::SnippetRepositoryRegistry id: 17028, retry_at: nil, last_synced_a...
irb(main):004:0> sr = srr.snippet_repository
=> #<SnippetRepository shard_id: 88, snippet_id: 1706247, disk_path: "@snip...
irb(main):005:0> pp srr
#<Geo::SnippetRepositoryRegistry:0x00007f92f8ca8798
 id: 17028,
 retry_at: nil,
 last_synced_at: Tue, 27 Apr 2021 17:57:41.267460000 UTC +00:00,
 created_at: Tue, 27 Apr 2021 16:38:21.179499000 UTC +00:00,
 snippet_repository_id: 1706247,
 state: 2,
 retry_count: 0,
 last_sync_failure: nil,
 force_to_redownload: false,
 missing_on_primary: false,
 verification_started_at: Sat, 12 Jun 2021 00:05:13.154482000 UTC +00:00,
 verified_at: Sat, 12 Jun 2021 00:05:15.302098000 UTC +00:00,
 verification_retry_at: Sat, 12 Jun 2021 01:05:59.302057000 UTC +00:00,
 verification_retry_count: 1069,
 verification_state: 3,
 checksum_mismatch: true,
 verification_checksum: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a",
 verification_checksum_mismatched: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a",
 verification_failure:
  "Checksum does not match the primary checksum {:checksum=>\"d20067b10c938e97c57680ceefd3d2d9a1a12d7a\", :primary_checksum=>\"668cf873ec07a2bad5b0c736eaf454837be01095\"}">
=> #<Geo::SnippetRepositoryRegistry id: 17028, retry_at: nil, last_synced_at: "2021-04-27 17:57:41.267460000 +0000", created_at: "2021-04-27 16:38:21.179499000 +0000", snippet_repository_id: 1706247, state: 2, retry_count: 0, last_sync_failure: nil, force_to_redownload: false, missing_on_primary: false, verification_started_at: "2021-06-12 00:05:13.154482000 +0000", verified_at: "2021-06-12 00:05:15.302098000 +0000", verification_retry_at: "2021-06-12 01:05:59.302057000 +0000", verification_retry_count: 1069, verification_state: 3, checksum_mismatch: true, verification_checksum: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_checksum_mismatched: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_failure: "Checksum does not match the primary checksum {:che...">
irb(main):006:0> pp sr
#<SnippetRepository:0x00007f930b3bc6d0
 shard_id: 88,
 snippet_id: 1706247,
 disk_path:
  "@snippets/94/0c/940cb63f6e47d647568e99b3987871f4dde8151e1fda9751392bea5899fe0570",
 verification_retry_count: 0,
 verification_retry_at: nil,
 verified_at: Wed, 14 Apr 2021 17:18:47.611491000 UTC +00:00,
 verification_checksum: "668cf873ec07a2bad5b0c736eaf454837be01095",
 verification_failure: nil,
 verification_state: 2,
 verification_started_at: Wed, 14 Apr 2021 17:18:47.273251000 UTC +00:00>
=> #<SnippetRepository shard_id: 88, snippet_id: 1706247, disk_path: "@snippets/94/0c/940cb63f6e47d647568e99b3987871f4dd...", verification_retry_count: 0, verification_retry_at: nil, verified_at: "2021-04-14 17:18:47.611491000 +0000", verification_checksum: "668cf873ec07a2bad5b0c736eaf454837be01095", verification_failure: nil, verification_state: 2, verification_started_at: "2021-04-14 17:18:47.273251000 +0000">
irb(main):007:0> srr.replicator.sync_repository
=> true
irb(main):008:0> srr.reload
=> #<Geo::SnippetRepositoryRegistry id: 17028, retry_at: nil, last_synced_at: "2021-06-12 00:37:33.603111000 +0000", created_at: "2021-04-27 16:38:21.179499000 +0000", snippet_repository_id: 1706247, state: 2, retry_count: 0, last_sync_failure: nil, force_to_redownload: false, missing_on_primary: false, verification_started_at: "2021-06-12 00:05:13.154482000 +0000", verified_at: "2021-06-12 00:05:15.302098000 +0000", verification_retry_at: "2021-06-12 01:05:59.302057000 +0000", verification_retry_count: 1069, verification_state: 0, checksum_mismatch: true, verification_checksum: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_checksum_mismatched: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_failure: nil>
irb(main):009:0> srr.reload
=> #<Geo::SnippetRepositoryRegistry id: 17028, retry_at: nil, last_synced_at: "2021-06-12 00:37:33.603111000 +0000", created_at: "2021-04-27 16:38:21.179499000 +0000", snippet_repository_id: 1706247, state: 2, retry_count: 0, last_sync_failure: nil, force_to_redownload: false, missing_on_primary: false, verification_started_at: "2021-06-12 00:05:13.154482000 +0000", verified_at: "2021-06-12 00:05:15.302098000 +0000", verification_retry_at: "2021-06-12 01:05:59.302057000 +0000", verification_retry_count: 1069, verification_state: 0, checksum_mismatch: true, verification_checksum: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_checksum_mismatched: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_failure: nil>
irb(main):010:0> srr.replicator.verify
=> true
irb(main):011:0> srr.reload
=> #<Geo::SnippetRepositoryRegistry id: 17028, retry_at: nil, last_synced_at: "2021-06-12 00:37:33.603111000 +0000", created_at: "2021-04-27 16:38:21.179499000 +0000", snippet_repository_id: 1706247, state: 2, retry_count: 0, last_sync_failure: nil, force_to_redownload: false, missing_on_primary: false, verification_started_at: "2021-06-12 00:38:08.318221000 +0000", verified_at: "2021-06-12 00:38:08.344701000 +0000", verification_retry_at: "2021-06-12 01:39:00.344633000 +0000", verification_retry_count: 1070, verification_state: 3, checksum_mismatch: true, verification_checksum: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_checksum_mismatched: "d20067b10c938e97c57680ceefd3d2d9a1a12d7a", verification_failure: "Checksum does not match the primary checksum {:che...">
irb(main):012:0>