Gitaly: add runbook for DeleteRefs prepare failures on a repository

What

Adds docs/gitaly/gitaly-deleterefs-prepare-failure.md: a runbook for DeleteRefs failing with Internal / unable to prepare on one repository.

Why

Since 2026-09-25 two repositories on two production Gitaly nodes have produced a steady ~17,800 DeleteRefs/Internal errors a day (about a fifth to a quarter of all Gitaly server-fault errors on gitlab.com on recent days). The cause is visible in the Gitaly log line itself — json.error_metadata.stderr carries git's message, here a reference name conflict with a packed reference — but no runbook pointed at that field or said what to do, so the alarm sat for 11 days. Tracked in the confidential Gitaly SRE tracker: https://gitlab.com/gitlab-com/gl-infra/sandbox/gitaly-sre-flow/-/issues/1 (and #39 (closed) there, "Runbook gap", which this closes).

What the runbook covers

  • Symptoms and how to tell this apart from stale locks (DeleteRefs/Aborted, existing runbook) and from packed-refs locked contention.
  • Which log fields identify the repository, node and path, and which carries git's message.
  • A table of git message classes → meaning → remedy (name conflict, stale lock, packed-refs lock, corruption, in-batch conflict).
  • Read-only confirmation from a Rails console or via gitaly git on the node.
  • The remedy through the DeleteRefs RPC from a Rails console (no edits on disk — there is no replica), including the loose-children case, draining the pending BatchedGitRefUpdates::Deletion rows, and the Kibana verification.
  • A specific escalation path.

Related: a Gitaly MR makes the conflict visible in json.error instead of exit status 128, and a gitlab-org/gitlab issue asks for back-off in the caller; links are in the tracker issue.

Merge request reports

Loading
Loading