Loading
Let Geo container repository sync retries back off
What does this MR do and why?
Geo::ContainerRepositorySyncService#execute calls registry.pending! before
every attempt. The pending transition resets retry_count to 0 and
retry_at to nil, so a container repository that keeps failing never
accumulates a retry count: it fails, moves to failed with retry_count 1,
and the next attempt resets it to 0 before failing back to 1. Delay#delay is
evaluated at a count of 1 forever, so the row retries every 16 to 74 seconds
indefinitely and never widens toward the one-hour ceiling that binds every
other replicable from its eighth failure. For a repository that cannot
converge that is on the order of 1,900 attempts a day, each one billing the
primary a tag listing plus one digest request per tag before any transfer
starts.
Every other replicable moves to pending only while consuming an event.
Geo::RepositoryReplicatorStrategy and Geo::BlobReplicatorStrategy both do
this, Geo::SyncWorker's class comment says the scheduled syncs it runs do not
begin by marking the registry pending, and the replication-state diagram in
doc/development/geo.md draws
Failed to Started as a Retry edge, separate from the Failed to
Pending "Mark for resync" edge.
This MR moves the transition into
Geo::ContainerRepositoryReplicator#consume_event_updated, with the same race
comment the strategies carry. An update event arriving while a sync is running
still moves the row to pending, mark_synced_atomically still finds it out
of started, and mark_sync_as_successful still reschedules. A scheduled
retry now goes failed to started with no pending step, so the count
climbs and the delay widens.
A bulk Resync writes with update_all in the shared registry base, bypassing
the pending callback, so it no longer zeroes accumulated backoff for
container repositories. That now matches every other replicable, and a
per-registry Resync is unaffected.
References
Any container repository that cannot converge already retries this way, so this change does not depend on anything else. It clears the way for !253800, which reports per-tag sync failures and moves more repositories onto this retry path.
How to set up and validate locally
Specs cover the retry accumulation and the event-path transition. I also ran it on a two-site 19.2.1 Geo pair with the container registry on the metadata database, patching the secondary only.
- Replicate a single-tag image to the secondary, then break the tag: delete a
layer blob from the primary's object storage, restart the primary registry,
and on the secondary
DELETE /v2/<repo>/manifests/<digest>so the tag has to transfer again. - Let the secondary's scheduler retry on its own and read the tracking row
(
Geo::ContainerRepositoryRegistry) across several attempts.
Unpatched, retry_count stays at 1 and retry_at stays 16 to 74 seconds out
across every retry. Patched, retry_count climbs 1, 2, 3, 4 and retry_at
widens through the backoff bands (16 to 74, 31 to 118, 96 to 212, 271 to 416
seconds). An update event pushed while a sync is running still moves the row
to pending and reschedules. Repairing the tag converges the row to synced
with retry_count 0. One repository left broken and untouched across the
patch read retry_count 1 after nine failures before it and retry_count 5
at the matching backoff floor after it.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
AI-Generated Content Disclosure: This MR was prepared with assistance from Claude Code. The output has been reviewed for correctness, verified against source, and validated with RSpec and RuboCop on a GitLab source checkout.