Gitaly: add runbook for DeleteRefs prepare failures on a repository
What
Adds docs/gitaly/gitaly-deleterefs-prepare-failure.md: a runbook for DeleteRefs failing with Internal / unable to prepare on one repository.
Why
Since 2026-09-25 two repositories on two production Gitaly nodes have produced a steady ~17,800 DeleteRefs/Internal errors a day (about a fifth to a quarter of all Gitaly server-fault errors on gitlab.com on recent days). The cause is visible in the Gitaly log line itself — json.error_metadata.stderr carries git's message, here a reference name conflict with a packed reference — but no runbook pointed at that field or said what to do, so the alarm sat for 11 days. Tracked in the confidential Gitaly SRE tracker: https://gitlab.com/gitlab-com/gl-infra/sandbox/gitaly-sre-flow/-/issues/1 (and #39 (closed) there, "Runbook gap", which this closes).
What the runbook covers
- Symptoms and how to tell this apart from stale locks (
DeleteRefs/Aborted, existing runbook) and frompacked-refs lockedcontention. - Which log fields identify the repository, node and path, and which carries git's message.
- A table of git message classes → meaning → remedy (name conflict, stale lock, packed-refs lock, corruption, in-batch conflict).
- Read-only confirmation from a Rails console or via
gitaly giton the node. - The remedy through the
DeleteRefsRPC from a Rails console (no edits on disk — there is no replica), including the loose-children case, draining the pendingBatchedGitRefUpdates::Deletionrows, and the Kibana verification. - A specific escalation path.
Related: a Gitaly MR makes the conflict visible in json.error instead of exit status 128, and a gitlab-org/gitlab issue asks for back-off in the caller; links are in the tracker issue.