Geo verification_retry_count can never exceed 1, so backoff never engages

Summary

On a Geo secondary, verification_retry_count on a registry can never exceed 1, and checksum_mismatch is cleared before every re-verification.

Details

Two behaviours combine to cause it:

  • A verification failure calls failed in before_verification_failed, so the registry resyncs. While the sync state is failed the row is excluded from verification_failed_batch and needs_verification, because available_verifiables is synced. Verification therefore cannot retry until the resync completes.
  • The resync's :start transition disables verification (Geo::VerifiableRegistry#before_started), and after_synced then fires verification_pending! from verification_disabled, which hits the branch that clears the verification failure fields.

So the counter oscillates 1 -> 0 forever:

verification_state=verification_failed   sync_state=failed  checksum_mismatch=true   verification_retry_count=1
verification_state=verification_pending  sync_state=synced  checksum_mismatch=false  verification_retry_count=0
verification_state=verification_failed   sync_state=failed  checksum_mismatch=true   verification_retry_count=1
verification_state=verification_pending  sync_state=synced  checksum_mismatch=false  verification_retry_count=0

Impact

  • The progressive backoff of syncs-due-to-verification-failures that before_verification_failed documents never engages. A registry with a genuine checksum mismatch keeps resyncing at full speed instead of backing off, which wastes resources on both the primary and the secondary.
  • verification_retry_count is unusable as a "how many consecutive verification failures" signal. Detection of persistent checksum mismatches (!246765 (merged)) cannot fire until this is fixed.

Affected versions

Introduced by 08492a70, which disabled verification on the :start transition so that invalid verification_failed rows could no longer block resyncs (#562921 (closed)). That commit shipped in 19.1, so 19.1, 19.2 and 19.3 before the fix are affected.

19.0 and earlier are not affected. They have no before_started override, so start! leaves the row in verification_failed and after_synced takes the verification_failed => verification_pending branch that already preserves the counter.

How to reproduce

# On a secondary
registry = Geo::ProjectRepositoryRegistry.find_by(project_id: <id>)
allow(registry).to receive(:ready_to_verify?).and_return(true) # or use a project whose primary verification succeeded

3.times do
  registry.verification_failed_with_message!('Checksum does not match the primary checksum')
  registry.start!    # the resync scheduled by the verification failure
  registry.synced!
  puts registry.reload.verification_retry_count
end

This prints 1, 1, 1. It should print 1, 2, 3.

Fix

verification_disabled is included in the transition that preserves the retry fields, alongside verification_failed.

Edited by Douglas Barbosa Alexandre