Geo: Resync instead of fail when a checksum mismatch is transient

What does this MR do and why?

When a Geo secondary verifies a replicable, the primary may have updated the data (and its checksum) after the secondary last synced it. This can happen for example if an entity (Project Repository, for example) receives rapid updates faster than Geo can complete a sync.

The two checksums describe different versions of the data, so the mismatch is expected - but today it is recorded as a verification_failed with checksum_mismatch: true, which shows up red in the Geo UI and Grafana during high-activity periods (for example Dedicated migrations).

With this change, when the primary's verified_at is newer than the registry's last_synced_at, the registry schedules a quiet resync (pending) instead of recording a failure. Genuine mismatches - where the primary checksummed before our last sync - still fail exactly as before, so they stay visible to the self-heal reporting added in !246765 (merged).

Closes #556537 (closed)

References

• Issue: #556537 (closed)
• Related self-heal work: !246765 (merged) (persistent mismatches reported to the primary - transient mismatches no longer enter its scope)

Screenshots or screen recordings

Before	After

How to set up and validate locally

Requires a GDK Geo pair (primary + secondary).

1. Pick a project that is synced and verified on the secondary (secondary Rails console):

   reg = Geo::ProjectRepositoryRegistry.synced.verification_succeeded.first
   project_id = reg.project_id

2. Simulate the primary updating the repo after the secondary's last sync (primary Rails console) - this is the state a push leaves behind once the primary re-checksums:

   state = Project.find(project_id).project_state
   state.update_columns(verification_checksum: "aaaa" + state.verification_checksum[4..], verified_at: Time.current)

3. Watch the Geo log on the secondary (shell, from the secondary GDK's gitlab/ directory):

   tail -f log/geo.log | grep --line-buffered "Checksum mismatch"

4. Trigger verification on the secondary (secondary Rails console):

   reg.replicator.verify
   reg.reload
   reg.state_name            # => :pending   (was :failed before this MR)
   reg.verification_failure  # => nil        (was "Checksum does not match...")
   reg.checksum_mismatch     # => false      (was true)

   The tail from step 3 prints "Checksum mismatch caused by primary update after sync, resyncing" at INFO.

5. Clean up - restore the real checksum on the primary and watch the secondary heal itself:

   # primary console:
   Project.find(project_id).replicator.verify

   Within a couple of minutes the registry returns to synced + verification_succeeded with no manual intervention.

Optional: sustained churn simulation (the customer scenario)

Simulates a repo receiving updates faster than the secondary can verify - a real change (ref flip) every 3 seconds, with the full Geo event + re-checksum pipeline. Uses write_ref so it works regardless of license/push access.

Run on the primary (rails runner churn.rb, Ctrl-C to stop):

# churn.rb - flip a ref + re-checksum project 1 every 3s
project = Project.find(1) # any synced+verified project
repo = project.repository
head = repo.commit.sha
parent = repo.commit("#{head}~1").sha

i = 0
loop do
  i += 1
  repo.write_ref("refs/heads/geo-churn-demo", i.even? ? head : parent)
  repo.expire_branches_cache
  project.replicator.geo_handle_after_update
  project.replicator.verify
  puts "cycle=#{i} primary checksum=#{project.project_state.reload.verification_checksum[0, 8]}"
  sleep 3
end

Watch the registry on the secondary (it flickers synced↔️pending/started, never failed):

loop do
  r = Geo::ProjectRepositoryRegistry.find_by(project_id: 1).reload
  puts "#{Time.now.strftime('%H:%M:%S')} sync=#{r.state_name} verif=#{r.verification_state_name} mismatch=#{r.checksum_mismatch}"
  sleep 5
end

Without this MR, the same stimulus leaves the registry cycling through failed with checksum_mismatch: true and last_sync_failure: "Verification failed with: Checksum does not match...".

Cleanup: stop the loop, then on the primary: project.repository.raw.delete_refs("refs/heads/geo-churn-demo"); project.repository.expire_branches_cache; project.replicator.geo_handle_after_update; project.replicator.verify - the secondary resyncs and goes green on its own.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.