Repair a stray parent ref in batched Git ref deletions, back off on failure
What does this MR do and why?
BatchedGitRefUpdates::ProjectCleanupWorker runs every minute for every project with pending deletions. When a bare reference such as refs/merge-requests/<iid> exists, git refuses the whole batch that deletes refs/merge-requests/<iid>/head, so the same batch has failed every minute for weeks on the affected repositories until someone repairs them from a console (#632245). Gitaly now reports that case as a typed FailedPrecondition detail, DeleteRefsError.reference_conflict, naming both references (gitaly!9409 (merged)).
Two changes:
- Client mapping.
Gitlab::GitalyClient::RefService#delete_refsmaps the new detail toGitlab::Git::ReferenceConflictError, which carriesexisting_reference_nameandconflicting_reference_name. - Repair, behind a flag. With
batched_git_ref_updates_repair_stray_parentenabled (gitlab_com_derisk, rollout issue #634369),ProjectCleanupServicerepairs the repository the way the operator runbook (gitlab-com/runbooks!11733) does: keep the commit the parent points to around (Repository#keep_around, sourcebatched_git_ref_updates_repair), verify the keep-around reference, delete the parent, retry the batch once. A second conflict propagates. The repair only runs when the existing reference is a parent of the one being deleted; the inverse conflict (a reference someone pushed below one Rails is deleting, possible because pushes to internal namespaces are not blocked, #623362) is left alone and reported. Each repair is logged with the project id and both references.
Independently of the flag, back-off: a project whose batch fails is no longer retried every minute. Consecutive failures are counted in Redis (batched_git_ref_updates:project_cleanup:backoff:<project_id>, kept for the current delay plus one day) and the project is skipped until a retry time that doubles from one minute to one day; a successful run clears the record, and a failing run that deleted something first starts the delay again. The worker still raises, so the failure stays visible in Sidekiq; the scheduler's per-minute enqueues become no-ops inside the window.
Dependency and merge order
- gitaly!9409 (merged) (merged 2026-10-21) adds the detail. The first
gitalygem that carriesGitaly::DeleteRefsError#reference_conflictis 19.6.0.pre.rc2 (2026-10-29); the~> 19.5.0pin excludes it, so until the gem bump the client spec for:reference_conflictstubs the decoded detail (Gitlab::GitalyClient.decode_detailed_error), the wayoperation_service_spec.rbdoes forUserMergeToRefError. The arm is covered with the current gem; the stub is replaced bynew_detailed_errorwith the bump. Everything else runs against the current gem. - This MR can merge before the gem bump: with the current gem the
:reference_conflictarm is unreachable, a conflict still surfaces asInternaland the repair path is inert; the back-off applies to it regardless. The flag has no effect until the gem arrives (prerequisite on #634369).
How to set up and validate locally
bin/rspec spec/services/batched_git_ref_updates/project_cleanup_service_spec.rb spec/lib/gitlab/gitaly_client/ref_service_spec.rb- Repair:
refs/merge-requests/1is created on a fixture repository, a pending deletion ofrefs/merge-requests/1/headis queued, the firstdelete_refsof that batch raises the mapped error (the test Gitaly may predate !9409 (closed)), and the spec checks the keep-around reference exists, the parent is gone, the deletion is processed and the repair is logged; a second conflict propagates; an unresolvable parent is left alone; flag off leaves everything as before. - Back-off: a failing batch records
failures=1andretry_afterone minute out; the next run inside the window does not load the project and returnsskipped_until; the delay doubles per consecutive failure up to one day; the record outlives the longest delay; a failing run that deleted something first restarts at one minute; a successful run clears the record. - The same specs run against a Gitaly built from gitaly!9409 (merged) in a separate CI job (gitlab-com/gl-infra/sandbox/gitaly-sre-flow,
rails-spec); the three examples added in the review round were first run against the unchanged service and fail there, and the stubbed:reference_conflictexample was first run against the client without that arm and fails there.
MR acceptance checklist
- Fail-first tests for the repair and the back-off; flag-off path unchanged
- No customer repository or node named; figures are aggregate
- Reviewer roulette picks honoured (@radbatnag)
Related: gitaly!9409 (merged), #632245, gitlab-org/gitaly#7432, rollout #634369. Tracker: gitlab-com/gl-infra/sandbox/gitaly-sre-flow#79 (confidential).