## Summary The Geo checksum-mismatch self-heal (gitlab-org/gitlab!246765, `geo_self_heal_checksum_mismatch`) never heals anything on a secondary that has persistent checksum mismatches greater than `BATCH_SIZE` in all repository types. The secondary's report is larger than the primary allows, so the primary rejects every report. `Geo::ChecksumMismatchReportingService` collects up to `BATCH_SIZE = 100` candidates **per repository replicator type** (project, wiki, design, snippet, group wiki and container repositories) and sends them all in **one** request: ```ruby # ee/app/services/geo/checksum_mismatch_reporting_service.rb BATCH_SIZE = 100 def mismatches @mismatches ||= Gitlab::Geo.repository_replicator_classes.flat_map do |replicator_class| candidates_for(replicator_class) # up to BATCH_SIZE each end end ``` The primary's endpoint rejects any request with more than 100 entries: ```ruby # ee/lib/ee/api/geo.rb (POST /geo/failures) bad_request!('Too many failures') if params[:failures].size > ::Geo::VerificationFailureReportWorker::MAX_FAILURES_PER_REQUEST # ee/app/workers/geo/verification_failure_report_worker.rb MAX_FAILURES_PER_REQUEST = 100 ``` A single request can therefore carry up to 600 entries, against a limit of 100. ## Impact - Any secondary whose persistent mismatches add up to more than 100 across repository types never gets a report accepted. On the primary, `Geo::ChecksumMismatchSelfHealService` never runs, and there are no `Marked primary record as pending checksum re-verification` lines. - The dedup marker is only written on success (`mark_as_reported!` inside `if response`). After a rejection the same oversized batch is resent on every run of `Geo::ChecksumMismatchReportingWorker` (`*/10 * * * *`), so it loops indefinitely. - Secondaries with fewer than 100 candidates in total are not affected, which is probably why this wasn't noticed. The existing service spec stubs `repository_replicator_classes` to a single class. ## Observed in production (GitLab 19.3.2) On a secondary with a few thousand container repository mismatches and a few hundred project repository mismatches: - **Secondary:** every `Geo::ChecksumMismatchReportingWorker` run in a 7-day window (553 runs) finished `done`, and each logged from `Geo::ChecksumMismatchReportingService`: ``` Could not connect to Geo primary node - HTTP Status Code: 400 Bad Request 400 Bad request - Too many failures ``` - **Primary:** zero `Marked primary record as pending checksum re-verification` lines over 14 days. - The failed counts for `container_repositories` and `project_repositories` did not change apart from normal retries. ## Steps to reproduce 1. On a secondary running 19.3+, have more than 100 registries in total across two or more repository types with `checksum_mismatch: true` and `verification_retry_count >= checksum_mismatch_report_threshold` (default 3). 2. Wait for `Geo::ChecksumMismatchReportingWorker` to run. 3. The secondary logs `400 Bad request - Too many failures`, and the primary records nothing. A spec that reproduces it (both limits stubbed to 2, with two project and two container repository mismatches): ```ruby # frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::ChecksumMismatchReportingService, :geo, feature_category: :geo_replication do include ::EE::GeoHelpers let_it_be(:primary) { create(:geo_node, :primary) } let_it_be(:secondary) do create(:geo_node, checksum_mismatch_report_threshold: 3, checksum_mismatch_self_heal_cooldown_minutes: 60) end let(:limit) { 2 } subject(:service) { described_class.new(secondary) } before do stub_current_geo_node(secondary) stub_const("#{described_class}::BATCH_SIZE", limit) stub_const('Geo::VerificationFailureReportWorker::MAX_FAILURES_PER_REQUEST', limit) allow(Gitlab::Geo).to receive(:repository_replicator_classes) .and_return([Geo::ProjectRepositoryReplicator, Geo::ContainerRepositoryReplicator]) create_list(:geo_project_repository_registry, limit, :verification_failed, checksum_mismatch: true, verification_retry_count: 3, verification_checksum_mismatched: 'abc') create_list(:geo_container_repository_registry, limit, :verification_failed, checksum_mismatch: true, verification_retry_count: 3, verification_checksum_mismatched: 'def') end it 'never sends more failures in one request than the primary accepts' do sent = nil allow(Gitlab::HTTP).to receive(:perform_request) do |_method, _url, options| sent = options[:body][:failures] double(success?: true, parsed_response: { 'message' => 'OK' }) # rubocop:disable RSpec/VerifiedDoubles -- generic HTTP response double end service.execute # Fails: 4 entries are sent in one request; the primary accepts at most 2. expect(sent.size).to be <= Geo::VerificationFailureReportWorker::MAX_FAILURES_PER_REQUEST end end ``` ## Proposed fix Make the reporter respect the endpoint's limit. Either: - **send in chunks:** `mismatches.each_slice(MAX_FAILURES_PER_REQUEST)`, one request per slice, calling `mark_as_reported!` per accepted slice; or - **cap the total:** a single shared budget of `MAX_FAILURES_PER_REQUEST` spread across types (e.g. round-robin), so each run sends ≤ 100 entries and later runs pick up the rest through the dedup marker. Either way, `BATCH_SIZE` should be derived from `Geo::VerificationFailureReportWorker::MAX_FAILURES_PER_REQUEST` rather than set separately, so the two can't drift apart again. The service spec should cover more than one replicator type. It may also be worth logging an oversized rejection more clearly than `Could not connect to Geo primary node`, which reads like a connectivity problem. ## Workaround Raising `checksum_mismatch_report_threshold` on the secondary's Geo site shrinks the candidate set, and may bring the total under 100. That only works if the mismatched registries have differing `verification_retry_count` values.