Repair a stray parent ref in batched Git ref deletions, back off on failure

What does this MR do and why?

BatchedGitRefUpdates::ProjectCleanupWorker runs every minute for every project with pending deletions. When a bare reference such as refs/merge-requests/<iid> exists, git refuses the whole batch that deletes refs/merge-requests/<iid>/head, so the same batch has failed every minute for weeks on the affected repositories until someone repairs them from a console (#632245). Gitaly now reports that case as a typed FailedPrecondition detail, DeleteRefsError.reference_conflict, naming both references (gitaly!9409 (merged)).

Two changes:

  1. Client mapping. Gitlab::GitalyClient::RefService#delete_refs maps the new detail to Gitlab::Git::ReferenceConflictError, which carries existing_reference_name and conflicting_reference_name.
  2. Repair, behind a flag. With batched_git_ref_updates_repair_stray_parent enabled (gitlab_com_derisk, rollout issue #634369), ProjectCleanupService repairs the repository the way the operator runbook (gitlab-com/runbooks!11733) does: keep the commit the parent points to around (Repository#keep_around, source batched_git_ref_updates_repair), verify the keep-around reference, delete the parent, retry the batch once. A second conflict propagates. The repair only runs when the existing reference is a parent of the one being deleted; the inverse conflict (a reference someone pushed below one Rails is deleting, possible because pushes to internal namespaces are not blocked, #623362) is left alone and reported. Each repair is logged with the project id and both references.

Independently of the flag, back-off: a project whose batch fails is no longer retried every minute. Consecutive failures are counted in Redis (batched_git_ref_updates:project_cleanup:backoff:<project_id>, kept for the current delay plus one day) and the project is skipped until a retry time that doubles from one minute to one day; a successful run clears the record, and a failing run that deleted something first starts the delay again. The worker still raises, so the failure stays visible in Sidekiq; the scheduler's per-minute enqueues become no-ops inside the window.

Dependency and merge order

  • gitaly!9409 (merged) (merged 2026-10-21) adds the detail. The first gitaly gem that carries Gitaly::DeleteRefsError#reference_conflict is 19.6.0.pre.rc2 (2026-10-29); the ~> 19.5.0 pin excludes it, so until the gem bump the client spec for :reference_conflict stubs the decoded detail (Gitlab::GitalyClient.decode_detailed_error), the way operation_service_spec.rb does for UserMergeToRefError. The arm is covered with the current gem; the stub is replaced by new_detailed_error with the bump. Everything else runs against the current gem.
  • This MR can merge before the gem bump: with the current gem the :reference_conflict arm is unreachable, a conflict still surfaces as Internal and the repair path is inert; the back-off applies to it regardless. The flag has no effect until the gem arrives (prerequisite on #634369).

How to set up and validate locally

bin/rspec spec/services/batched_git_ref_updates/project_cleanup_service_spec.rb spec/lib/gitlab/gitaly_client/ref_service_spec.rb
  • Repair: refs/merge-requests/1 is created on a fixture repository, a pending deletion of refs/merge-requests/1/head is queued, the first delete_refs of that batch raises the mapped error (the test Gitaly may predate !9409 (closed)), and the spec checks the keep-around reference exists, the parent is gone, the deletion is processed and the repair is logged; a second conflict propagates; an unresolvable parent is left alone; flag off leaves everything as before.
  • Back-off: a failing batch records failures=1 and retry_after one minute out; the next run inside the window does not load the project and returns skipped_until; the delay doubles per consecutive failure up to one day; the record outlives the longest delay; a failing run that deleted something first restarts at one minute; a successful run clears the record.
  • The same specs run against a Gitaly built from gitaly!9409 (merged) in a separate CI job (gitlab-com/gl-infra/sandbox/gitaly-sre-flow, rails-spec); the three examples added in the review round were first run against the unchanged service and fail there, and the stubbed :reference_conflict example was first run against the client without that arm and fails there.

MR acceptance checklist

  • Fail-first tests for the repair and the back-off; flag-off path unchanged
  • No customer repository or node named; figures are aggregate
  • Reviewer roulette picks honoured (@radbatnag)

Related: gitaly!9409 (merged), #632245, gitlab-org/gitaly#7432, rollout #634369. Tracker: gitlab-com/gl-infra/sandbox/gitaly-sre-flow#79 (confidential).

Edited by Kranthi Erusu

Merge request reports

Loading
Loading