Geo: Add targeted container repository reverification task

Scott Murrayrequested to merge
591222-geo-reverify-container-repositories-since into master

What does this MR do and why?

When container repository replication is disabled on a Geo primary (via the geo_container_repository_replication ops feature flag), pushes to existing container repositories don't emit Geo::Event records, so secondary sites are never told to resync them. New repositories are backfilled automatically, but updated ones stay marked synced on secondaries until the periodic reverification cycle (default: every 7 days) eventually detects the checksum mismatch. The Geo admin UI only offers "Reverify all", which is a heavy hammer on large registries.

However, while replication is disabled, registry notifications still reach the primary, and — provided geo_container_repository_force_primary_checksumming remains enabled (the default) — the primary still re-checksums each pushed repository, updating verified_at on its container_repository_states record. This gives us a precise marker of which repositories changed during the outage window so that we know which ones to resync.

This is useful particularly in Dedicated cutover situations (when a customer is moving from self managed to Dedicated) where a cutover deadline is looming meaning there may not be enough time to simply "resync everything".

This MR adds a targeted recovery rake task that exploits the verified_at marker:

sudo gitlab-rake "gitlab:geo:reverify_container_repositories_since[7]"

It synchronously marks for reverification only the container repositories the primary verified in the last N days (default 7), then reports how many were marked. The periodic verification workers re-checksum them; the updated checksums propagate and secondaries detect the mismatch and resync only the affected repositories.

The marking uses the "slow iteration" batching pattern: the table is iterated in primary-key batches (filter-free boundary queries), and the filters are applied only within each PK-bounded batch, so every statement stays cheap regardless of data distribution. No background job or cursor is needed — re-running after an interruption resumes naturally, because already-marked rows become verification_pending and fall out of the scope.

A DRY_RUN=true mode reports how many repositories would be marked without changing anything, using the same batched iteration.

Changes:

  • New verified_after scope on Geo::VerificationStateDefinition
  • New Geo::ReverifyContainerRepositoriesService: synchronous, batched (slow iteration), with execute and dry_run modes
  • New rake task gitlab:geo:reverify_container_repositories_since[days] (primary-only, licence-gated, validates the days argument)
  • Documentation for the recovery procedure in the Geo container registry replication docs

No new index is added — see the database review section for the rationale and query plans.

Relates to #591222 (closed)

Changelog: added EE: true

Database review

Database review notes:

  • Addresses the previous review round: the earlier design applied the verified_at >= ? and verification_state != 0 filters in the each_batch boundary queries, which had no index coverage (worst case: scan the entire PK with heap lookups). Per the reviewer's suggestion, this now uses slow iteration: iteration runs over the bare table on the primary key only, and both filters are applied inside each PK-bounded batch.
  • Consequently the boundary queries (1, 2, 6 below) carry no filters at all — they are pure PK index-only scans whose cost is independent of data distribution by construction. The filtered statements (3, 4, 5) are always bounded to a ≤1,000-row PK range; the measured worst case (a batch where nothing matches the filter) is 457 buffers. Adverse data distributions can only increase the number of cheap batches, never the cost of any single statement.
  • No new index is added for verified_at: it is rewritten on every successful verification (one of the hottest-written columns on this table), so a dedicated index would tax every checksum cycle to serve a rarely-run, manually-invoked recovery rake task. With slow iteration, no statement needs it.
  • The task runs synchronously in the rake process, one batch at a time (BATCH_SIZE = 1_000) — no Sidekiq, no Redis cursor. Batches commit independently.
  • Geo is not active on GitLab.com (only GitLab Dedicated and Self-Managed), so container_repository_states has negligible production data in the postgres.ai snapshot (~560k rows, all verification_state = 0 / verified_at IS NULL).
  • We therefore seeded 1,000,000 rows (plus 1M parent container_repositories rows to satisfy the FK) with a deliberately adversarial distribution — addressing the previous round's observation that favourable data distribution can flatter plans: verification states interleaved across the PK via modulo (85% verification_state = 2, 5% = 3, 10% = 0 — no contiguous state ranges), and verified_at decorrelated from the PK via random() over a 30-day window, so a 7-day cutoff matches ~23% of non-pending rows scattered across the whole table. ANALYZE container_repository_states was run after seeding, before capturing plans.

Query 1: first batch boundary (once per task run)

each_batch opening boundary. No filters — pure PK index-only scan.

SELECT "container_repository_states"."container_repository_id"
FROM "container_repository_states"
ORDER BY "container_repository_states"."container_repository_id" ASC
LIMIT 1;

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54011/commands/156389 — 1.9ms, 4 buffers, PK index-only scan, no filter

Query 2: per-batch boundary lookup (once per 1,000-row batch)

The query shape flagged in the previous review round — now filter-free. Reads exactly BATCH_SIZE + 1 index entries regardless of data distribution.

SELECT "container_repository_states"."container_repository_id"
FROM "container_repository_states"
WHERE "container_repository_states"."container_repository_id" >= 1000000000
ORDER BY "container_repository_states"."container_repository_id" ASC
LIMIT 1 OFFSET 1000;

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54011/commands/156390 — 0.8ms, 9 buffers, PK index-only scan, no filter

Query 3: per-batch COUNT (dry run, once per batch)

Both filters applied inside the PK-bounded batch. 214 of 1,000 rows matched — the interleaved seeding at work.

SELECT COUNT(*)
FROM "container_repository_states"
WHERE "container_repository_states"."container_repository_id" >= 1000500000
  AND "container_repository_states"."container_repository_id" < 1000501000
  AND "container_repository_states"."verification_state" != 0
  AND "container_repository_states"."verified_at" >= '2026-07-16 17:00:00';

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54011/commands/156391 — 1.2ms, 20 buffers, bounded PK index scan

Query 4: per-batch UPDATE (once per batch)

UPDATE "container_repository_states"
SET "verification_state" = 0
WHERE "container_repository_states"."container_repository_id" >= 1000500000
  AND "container_repository_states"."container_repository_id" < 1000501000
  AND "container_repository_states"."verification_state" != 0
  AND "container_repository_states"."verified_at" >= '2026-07-16 17:00:00';

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54011/commands/156392 — 4.9ms, 214 rows updated, ~134KB WAL, bounded PK index scan

Query 5: worst case — zero-match batch

Same COUNT with a future cutoff so the filter matches nothing: every row in the batch requires a heap visit and is removed by the filter. This is the per-statement ceiling — still bounded by the 1,000-row PK range.

SELECT COUNT(*)
FROM "container_repository_states"
WHERE "container_repository_states"."container_repository_id" >= 1000500000
  AND "container_repository_states"."container_repository_id" < 1000501000
  AND "container_repository_states"."verification_state" != 0
  AND "container_repository_states"."verified_at" >= '2026-08-01 00:00:00';

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54011/commands/156393 — 1.4ms, 457 buffers, 0 rows matched (1,000 removed by filter)

Query 6: worst case — final batch at the table tail

Boundary lookup where fewer than BATCH_SIZE rows remain: scans the remaining entries and stops.

SELECT "container_repository_states"."container_repository_id"
FROM "container_repository_states"
WHERE "container_repository_states"."container_repository_id" >= 1000999500
ORDER BY "container_repository_states"."container_repository_id" ASC
LIMIT 1 OFFSET 1000;

Plan: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54011/commands/156394 — 0.8ms, 9 buffers, PK index-only scan, no filter

References

Screenshots or screen recordings

Before After

How to set up and validate locally

Requires a GDK with Geo primary configured and the container registry enabled (registry.enabled: true in gdk.yml). You only need the primary — the whole change is primary-side.

1. Configure registry notifications (if not already set up)

GDK doesn't configure these by default. Append to <gdk-root>/registry/config.yml:

notifications:
  endpoints:
    - name: geo_event
      url: http://127.0.0.1:3000/api/v4/container_registry_event/events
      timeout: 2s
      threshold: 5
      backoff: 1s
      headers:
        Authorization: [notifications_secret]

The notifications_secret value must match registry.notification_secret in gitlab/config/gitlab.yml (GDK's default is literally notifications_secret). Then gdk restart registry.

2. Create a repo and push an image (baseline)

docker login 127.0.0.1:5100 -u root -p <PAT with api,read_registry,write_registry>
docker pull alpine:latest
docker tag alpine:latest 127.0.0.1:5100/root/<project-path>:v1
docker push 127.0.0.1:5100/root/<project-path>:v1

In rails console, force a checksum so the record has a baseline verified_at:

cr = ContainerRepository.last
cr.replicator.verify
st = Geo::ContainerRepositoryState.find_by(container_repository_id: cr.id)
st.verification_state_name # => :verification_succeeded
st.verified_at             # note this timestamp

3. Simulate the outage scenario

Feature.disable(:geo_container_repository_replication)
# force-checksumming must stay enabled (it's on by default):
Geo::ContainerRepositoryReplicator.verification_enabled? # => true
events_before = Geo::Event.where(replicable_name: 'container_repository').count

Push an updated image (must have new content, e.g. retag a different image as :v2) and wait ~30s for the notification to process. Note: Flipper caches flag state for ~1 min in running processes — wait a couple of minutes after disabling the flag before pushing, or the push may still emit an event.

Confirm the issue's premise:

# No replication event was emitted (this is the bug being recovered from):
Geo::Event.where(replicable_name: 'container_repository').count # == events_before

# But the push still marked the record for re-checksumming:
st.reload.verification_state_name # => :verification_pending

# Re-checksum (or wait for the periodic VerificationBatchWorker):
Geo::ContainerRepositoryReplicator.verify_batch
st.reload.verified_at # => advanced past the baseline timestamp

4. Validate the rake task

The task runs synchronously and reports the count when done — no Sidekiq involvement.

# Dry run — counts without changing anything:
DRY_RUN=true bundle exec rake "gitlab:geo:reverify_container_repositories_since[1]"
# => DRY RUN: 1 container repositories would be marked for reverification. (verified since <cutoff>)

# Real run:
bundle exec rake "gitlab:geo:reverify_container_repositories_since[1]"
# => Marked 1 container repositories for reverification. (verified since <cutoff>)

Confirm the targeted record was reset while others were untouched:

st.reload.verification_state_name # => :verification_pending

Re-running the task immediately reports Marked 0 ... — already-marked rows are pending and fall out of the scope, which is what makes an interrupted run safely resumable. (If you wait long enough for the periodic verification workers to re-checksum the record first, it legitimately re-enters the scope with a fresh verified_at, and re-marking it is harmless.)

5. Error paths (optional)

bundle exec rake "gitlab:geo:reverify_container_repositories_since[0]"   # => aborts: Days must be a positive integer
bundle exec rake "gitlab:geo:reverify_container_repositories_since[abc]" # => aborts: Days must be a positive integer

On a secondary GDK the task aborts with This command is only available on a primary node.

Cleanup

Feature.enable(:geo_container_repository_replication)

Revert the registry/config.yml change if you don't want notifications enabled permanently.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #591222 (closed)

Edited by Scott Murray

Merge request reports