Geo: Self-heal persistent checksum mismatches

Douglas Barbosa Alexandrerequested to merge
562796-geo-verification-failure-self-heal into master

What does this MR do?

Adds an automatic self-heal loop for Geo checksum mismatches that never resolve because the primary's stored checksum is stale rather than the secondary's copy being wrong. Today this requires manual intervention (Rails console or the admin data-management API) per resource.

  • A secondary that fails checksum verification for the same resource checksum_mismatch_report_threshold times in a row reports it to the primary (Redis-deduplicated, at most once per checksum_mismatch_self_heal_cooldown_minutes).
  • The primary receives these reports on a new generic POST /api/v4/geo/failures endpoint (deliberately not checksum-specific, since REST API v4 endpoints are expensive to change once shipped) and, if the record is still eligible (its current checksum still matches what the secondary saw, and it isn't within its own cooldown window), flips it back to verification_pending.
  • The existing verification pipeline then recomputes the checksum and propagates it to secondaries exactly as it does today — that part was already implemented and is untouched by this MR.

Both thresholds are configurable per Geo node (geo_nodes columns, exposed via the geo_nodes/geo_sites REST APIs and GraphQL, following the existing minimum_reverification_interval pattern).

Everything is behind the geo_self_heal_checksum_mismatch ops flag, disabled by default.

Follow-ups / open questions

Prerequisite for enabling the flag: !248366 (merged) must land first. verification_retry_count is currently reset to 0 (and checksum_mismatch back to false) by the resync that every verification failure triggers, so the checksum_mismatch + verification_retry_count >= threshold scope here cannot match anything until that is fixed. Details in that MR and in the discussion below.

  • Default threshold (3) and cooldown (60 min) values are placeholders — need to be validated against real-world retry cadence before enabling broadly.
  • High-traffic replicables (frequent commits to a project repository) can fail verification because the secondary cannot catch up, not because of corruption — #556537. Fixing it there stops those mismatches from ever being recorded as verification failures, so they never reach this scan's scope; the primary-side eligible? checksum comparison is a second guard in the meantime. Revisit here only if 556537 does not land.
  • Whether secondary-side reporting should live in its own cron worker (as implemented here) or fold into the existing Geo::MetricsUpdateWorker cycle.
  • Once #548532 (typed Geo error classes) lands, swap the error_type string / checksum_mismatch boolean check for the typed Geo::Errors::ChecksumMismatchError — no wire-format change needed since the string values are already aligned.
  • Once #602803 (Geo cleanup rake tasks) lands, consider whether the self-heal logic should be shared with its manual Geo::Tools::Resolutions.

Database

Raw SQL and query plan for the new persistent_checksum_mismatches scope (ee/app/models/concerns/geo/verifiable_registry.rb), used via .order(:verification_retry_at).limit(BATCH_SIZE * CANDIDATE_MULTIPLIER) in Geo::ChecksumMismatchReportingService#candidates_for. This scope only ever runs against repository replicator registries (Gitlab::Geo.repository_replicator_classes) — persistent stale-primary-checksum mismatches are a phenomenon specific to git-repository checksums, not blob checksums, so the reporting service only scans those registry tables: project_repository_registry, project_wiki_repository_registry, design_management_repository_registry, group_wiki_repository_registry, snippet_repository_registry.

Raw SQL (generated for Geo::ProjectRepositoryRegistry, representative of the other repository registry tables — identical shape):

SELECT "project_repository_registry".*
FROM "project_repository_registry"
WHERE "project_repository_registry"."checksum_mismatch" = TRUE
  AND "project_repository_registry"."verification_retry_count" >= 3
ORDER BY "project_repository_registry"."verification_retry_at" ASC
LIMIT 300

Caveat on scale: registry tables live in each Geo secondary's own tracking database, not gitlab.com's main production database, so Database Lab/postgres.ai doesn't cover them and I don't have access to a GitLab.com-scale Geo tracking database. To get a real (not fabricated) plan, I seeded a synthetic 2,000,000-row project_repository_registry locally and ran EXPLAIN (ANALYZE, BUFFERS). This shows the pattern, not a substitute for a plan pulled from an actual secondary's tracking DB — flagging this explicitly for whoever picks up the database review.

I initially assumed checksum_mismatch = true rows would have state = 2 (synced), verification_state = 3 (failed), matching the existing ..._failed_verification partial index's predicate. That's wrong: before_verification_failed (ee/app/models/concerns/geo/verifiable_registry.rb) unconditionally fires the sync state event failed whenever verification fails (ee/app/models/concerns/geo/replicable_registry.rb's event :failed do transition [:pending, :started, :synced, :failed] => :failed end covers every prior state), so a mismatched row is actually state = 3, verification_state = 3 — confirmed both by reading the transition table and empirically against a local tracking DB. The existing partial index can never match these rows, which is exactly why the plan below shows a sequential scan despite that index existing.

Plan without a checksum_mismatch-covering index:

Limit  (cost=32039.41..32059.25 rows=170 width=210) (actual time=30.838..32.850 rows=300 loops=1)
  Buffers: shared hit=20696
  ->  Gather Merge  (cost=32039.41..32059.25 rows=170 width=210) (actual time=30.837..32.832 rows=300 loops=1)
        Workers Planned: 2
        Workers Launched: 2
        ->  Sort  (cost=31039.39..31039.60 rows=85 width=210) (actual time=29.085..29.094 rows=239 loops=3)
              Sort Key: verification_retry_at
              Sort Method: top-N heapsort  Memory: 91kB
              ->  Parallel Seq Scan on project_repository_registry  (cost=0.00..31036.67 rows=85 width=210) (actual time=0.012..28.861 rows=3353 loops=3)
                    Filter: (checksum_mismatch AND (verification_retry_count >= 3))
                    Rows Removed by Filter: 663313
Planning Time: 0.079 ms
Execution Time: 32.876 ms

Given the reporting worker runs every minute (ee/config/schedule.yml: geo_checksum_mismatch_reporting_worker, cron */1 * * * *), this MR now adds a dedicated partial index per repository registry table (ee/db/geo/post_migrate/20260728120000..20260728120004):

add_concurrent_index :project_repository_registry, :verification_retry_at,
  where: 'checksum_mismatch = true', name: 'index_project_repository_registry_checksum_mismatch'

Plan with the new index:

Limit  (cost=0.29..9166.77 rows=193 width=210) (actual time=0.011..0.279 rows=300 loops=1)
  Buffers: shared hit=363 read=2
  ->  Index Scan using index_project_repository_registry_checksum_mismatch on project_repository_registry  (cost=0.29..9166.77 rows=193 width=210) (actual time=0.011..0.270 rows=300 loops=1)
        Filter: (verification_retry_count >= 3)
        Rows Removed by Filter: 63
Planning Time: 0.227 ms
Execution Time: 0.290 ms

~113x fewer buffer reads (365 vs 20,696) and ~113x faster (0.29ms vs 32.9ms) in this local test. Migrated and rolled back locally to confirm reversibility.

References

#562796 (closed)

Screenshots or screen recordings

Not applicable (backend-only change).

How to set up and validate locally

  1. Enable the flag: Feature.enable(:geo_self_heal_checksum_mismatch)

  2. Pick a project that has already synced and verified successfully on the secondary (project.verification_succeeded? is true). project_states.verification_checksum (the primary's stored repository checksum, delegated onto Project) is a regular main-database table, replicated to secondaries via normal PostgreSQL streaming replication — unlike the git data itself, no Geo-specific sync is involved, so corrupting it on the primary is instantly visible to secondaries.

    On the primary, corrupt the stored checksum without touching the repository itself, to simulate the primary's checksum going stale relative to its (unchanged) real data:

    project = Project.find_by_full_path('group/project')
project.verification_succeeded? # => true
project.update!(verification_checksum: project.verification_checksum.reverse) # still valid-looking hex, now wrong

    On the secondary, force verification attempts directly instead of waiting for the cron cadence — each call computes the secondary's real (unchanged, correct) checksum and compares it against the now-corrupted primary value, so it fails every time:

    replicator = Geo::ProjectRepositoryReplicator.new(model_record: Project.find_by_full_path('group/project'))
Gitlab::Geo.current_node.checksum_mismatch_report_threshold.times { replicator.verify }

registry = Geo::ProjectRepositoryRegistry.find_by(project_id: project.id)
registry.checksum_mismatch # => true
registry.verification_retry_count # => >= checksum_mismatch_report_threshold

    Then trigger the reporting worker directly rather than waiting up to a minute for its cron:

    Geo::ChecksumMismatchReportingWorker.new.perform

  3. Confirm the primary flips the resource to verification_pending and the secondary eventually reports verification_succeeded.

Automated coverage: ee/spec/services/geo/checksum_mismatch_reporting_service_spec.rb, ee/spec/services/geo/checksum_mismatch_self_heal_service_spec.rb, ee/spec/workers/geo/checksum_mismatch_reporting_worker_spec.rb, ee/spec/workers/geo/verification_failure_report_worker_spec.rb, ee/spec/requests/api/geo_spec.rb.

Edited by Douglas Barbosa Alexandre

Merge request reports