Resolve "Geo checksum-mismatch self-heal never runs when mismatches span multiple repository types (reporter exceeds MAX_FAILURES_PER_REQUEST)"

What does this MR do and why?

Batches the failures that the secondary sends to the primary for the checksum mismatch self-healing. Previously, all failures across all data types were sent in one request. Since the primary rejects more than 100 failures, then hosts with too many combined failures would never heal.

References

#631787 (closed)

How to set up and validate locally

  1. On a dev/test instance running master with Geo enabled, cause > 100 checksum failures across data types (or patch the persistent_checksum_mismatches scope to return > 100 objects.
  2. Observe the primary logs will contain errors about "Too many failures"
  3. Switch to this branch, the error messages should stop.
  4. Geo::VerificationFailureReportWorker should begin to clear issues. (If you patched the scope, I would expect the primary to begin logging "Unknown verification failure error_type, skipping")

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #631787 (closed)

Edited by Ian Baum

Merge request reports

Loading
Loading