Draft: Report per-tag sync failures in Geo container repository sync

What does this MR do and why?

Geo::ContainerRepositorySync#execute rescues a per-tag error, logs it, and returns true regardless, so its caller marks the sync successful. A repository whose tags all failed to transfer still reads synced, with last_sync_failure nil and nothing to return it to needs_sync_again, so a secondary can report healthy over an empty registry.

This collects each failure and, after both loops, raises Geo::Errors::ContainerRepositorySyncError. The caller already rescues into fail_registry_sync!, so the row moves to failed carrying the counts and tag names, then retries until it converges.

I raise after the loops, not inside the per-tag rescue, which was tried and reverted: asked for in #526306 (closed), implemented in !185424 (merged), which re-raised inside the rescue and left later tags unreachable, caught by #549520 (closed), reverted by !195515 (merged). Every tag is still attempted and still gets its own log line.

Expect this to surface a backlog. Rows reading synced today while swallowing failures move to failed on their next sync: correct reporting, but it will read as a regression on a dashboard.

Two choices you may want to redirect

One failed tag fails the whole repository. STATE_VALUES holds no partial state, and adding a fifth is a framework change across every replicable. tag_list_digest covers the whole tag list, so one missing tag already makes the repository unverifiable. failed is the only state feeding needs_sync_again, and retry is convergent. The counts carry what a fifth state would: 3 of 200 tags failed to sync differs from 200 of 200.

The message is built in the error class. The alternative, the shape eakca1 sketched on the issue, formats it in fail_registry_sync! from a structured payload. I kept it here so the 200 character cap sits beside the constants enforcing it and is testable without a registry. Either split works.

Gitlab::ErrorTracking now receives the aggregate class, so Sentry groups on it; the per-tag classes stay in geo.log.

References

Related to #577445, which has a second root cause, stale primary checksums, that this does not address.

How to set up and validate locally

Specs cover the message building and the registry outcome. I also ran the fix on a two-site 19.2.1 Geo pair with the container registry on the metadata database, patching the secondary only.

1. Replicate a single-tag image, then make the tag unrecoverable: delete one layer blob for it from the primary's object storage and restart the primary registry, which caches blob descriptors and otherwise keeps answering HEAD for the deleted blob. On the secondary, remove the tag so it has to transfer again; with the metadata database, emptying the bucket is not enough because the tag list lives in PostgreSQL, so also DELETE /v2/<repo>/manifests/<digest> on the secondary's registry.

2. From the secondary's Rails console:

   cr = ContainerRepository.find(<id>)
   r = Geo::ContainerRepositoryRegistry.find_by(container_repository_id: cr.id)
   r.replicator.resync
   r.reload.slice(:state, :retry_count, :last_sync_failure, :last_synced_at, :retry_at)

Unpatched, the row reads state 2, retry_count 0, last_sync_failure nil, with zero objects in the secondary's storage. Patched, it reads state 3, retry_count 1, retry_at 16 to 74 seconds out, and last_sync_failure Container repository sync failed: 1 of 1 tag failed to sync: v1; geo.log carries the existing per-tag line plus a Geo::ContainerRepositorySyncService line whose error field is that summary, and the scheduler retries on its own. Restoring the blob and restarting the primary registry converges the row back to state 2 with the failure cleared, and the next verification cycle reaches verification_state 2. With three tags and two broken, the message reads 2 of 3 tags failed to sync: t1, t2 and the intact tag lands.

retry_count stays at 1 and the delay stays in that band because Geo::ContainerRepositorySyncService#execute calls registry.pending! before every attempt, which resets the count. That is pre-existing and unchanged here.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

---

AI-Generated Content Disclosure: This MR was prepared with assistance from Claude Code. The output has been reviewed for correctness, verified against source, and validated with RSpec and RuboCop on a GitLab source checkout.