Orphaned ref lockfiles after a canceled reference transaction freeze replication until manually removed

Summary

When a reference transaction gets canceled while it is in the prepare phase, the git child is killed while it still holds ref lockfiles (packed-refs.lock or refs/**/*.lock). Those lockfiles stay on disk. From that point on every ReplicateRepository on the affected replica fails with cannot lock ref ... File exists (exit status 253) and the replica is stuck on its old generation.

The only thing that removes such locks is CleanStaleData with its 1h ReferenceLockfileGracePeriod, and that only runs as part of housekeeping. A replica whose only remaining traffic is the failing replication retry loop never triggers housekeeping, so the lock stays forever. In our latest incident nothing had cleaned it up almost 3 hours in and the generation gap was still growing. We had to delete the lockfiles by hand.

We run a 3-node Gitaly Cluster and have hit this three times in two months, each time through a different RPC (DeleteRefs, merge train ref updates, WriteRef). The trigger differs but the failure afterwards is always the same.

Environment

  • GitLab EE Omnibus 19.2.1 (first two incidents on 19.0.x), git 2.55.0-rc1
  • Praefect with 3 Gitaly storages, replication factor 3
  • Affected repos are the busy ones (a GitOps repo with frequent merges, merge trains, lots of pipelines)

Incident 1 (2026-06-05): DeleteRefs

A cron-fanned batch of MergeRequestCleanupRefsWorker jobs ran at least 5 concurrent /gitaly.RefService/DeleteRefs against the same repo. The losers waited 10s on packed-refs.lock (command.real_time_ms matches core.packedRefsTimeout), then died with unable to prepare: signal: killed: context canceled while holding the lock.

That part is the known #5368 (closed) pattern. gitlab!125109 (merged) serializes per merge request, so it does not help against cleanup jobs for different MRs of the same repo. We put a Gitaly concurrency limit on DeleteRefs (max_per_repo: 1) and that herd has not come back since.

Incident 2 (2026-06-30): merge train

An aborted merge train left locks on refs/merge-requests/N/{head,train,merge,rebase_on_merge}.lock plus packed-refs.lock, all with the same mtime. Same frozen replication afterwards.

Incident 3 (2026-08-05, 19.2.1): concurrent WriteRef on a keep-around ref

One merge event fanned out 4 MergeRequests::KeepAroundRefsWorker jobs within 570ms (same correlation ID). Each writes the same refs/keep-around/<sha> via /gitaly.RepositoryService/WriteRef. The duplicates collided in voting, Praefect canceled the transaction and update-ref was killed holding the loose ref lock:

07:26:50.710 WriteRef  grpc.code=Aborted   error="The operation could not be completed. Please try again."
             error_metadata: reference is locked already, reference=refs/keep-around/<sha>
07:26:50.743 ReferenceTransactionHook  error="vote failed: rpc error: code = Canceled desc = transaction has been canceled"  transaction.id=4226
07:26:50.753 WriteRef  grpc.code=Canceled  error="error when running update-ref command: signal: killed: context canceled"

refs/keep-around/<sha>.lock was left behind on both secondaries. The primary committed and moved on. Note there is no 10s contention timeout involved here, the cancel came about 1s in. Any canceled prepare can do this.

From 07:26:57 replication failed every few seconds on both secondaries:

ReplicateRepository ... replicating repository: synchronizing references:
fetch internal remote: exit status 253
stderr: cannot lock ref 'refs/keep-around/<sha>': Unable to create
'.../refs/keep-around/<sha>.lock': File exists

praefect metadata about 3 hours later, gap still growing:

Generation: 372914 (primary)
- storage-1: Generation: 372738, behind by 176 changes, Healthy: true, Valid Primary: false
- storage-2: Generation: 372738, behind by 176 changes, Healthy: true, Valid Primary: false
- storage-3: Generation: 372914, fully up to date,      Healthy: true, Valid Primary: true

Why it never self-heals

In incident 1 the locks were reaped 60m21s after creation, which is exactly ReferenceLockfileGracePeriod, because housekeeping happened to run on that repo. In incident 3 the gitaly log shows the last scheduled housekeeping for the repo at 05:17 and not a single removed files event until we cleaned up manually at 10:13. The lock was born 07:26. So for 2h47m nothing even tried.

That makes sense once you see the dependency: CleanStaleData runs via housekeeping, housekeeping is triggered by repository activity, and the only activity left on a frozen replica is the replication that the lock itself blocks. The cleanup depends on the operation it is supposed to unblock.

Impact

  • Replica frozen for hours until an admin deletes the lockfiles. After that replication heals within a minute.
  • The frozen replicas keep reporting Healthy: true. During the earlier incidents users saw stale reads: freshly pushed branches 404, MRs showing "no changes" and getting auto-closed. The read distribution docs say outdated replicas are excluded from reads. That did not hold up here, or something else routed those reads. Happy to dig out more evidence if that part is interesting.

Workarounds we use

  • max_per_repo: 1 on DeleteRefs. Covers incident 1 only, incident 3 came through WriteRef anyway. We may evaluate the same limit for WriteRef.
  • As a one-off recovery, deleting the *.lock files with an old mtime on the stuck replica works, replication recovers on the next retry. That is how we resolved the incidents so far.

Suggestions

  1. Clean up ref lockfiles when a reference transaction is canceled or aborted, also on the non-WAL path. #5595 punts this to "once transactions are on by default", but transactions do not run under Praefect at all right now, so cluster users never reach that fix.
  2. Or run the ref-lock part of CleanStaleData when ReplicateRepository keeps failing with cannot lock ref. The replication worker already knows the exact ref that is stuck.
  3. Make ReferenceLockfileGracePeriod configurable.
  4. On the Praefect side, a replica that is behind and has been failing replication for X minutes should not count as plain healthy. And the read routing exclusion for outdated replicas deserves a check for this state.

Related

  • #5595 (killed commands leave temp files behind, blocked on WAL by default, which cluster users cannot run)
  • #5368 (closed) / gitlab!125109 (merged) (DeleteRefs deadlock, fix only serializes within one MR)
  • #6470 (same "packed-refs locked" error, only proposes a status code change)
  • !6238 (merged) (removes stale packed-refs locks, but only at TransactionManager startup on the WAL path)
  • gitlab#591291 (Rails swallows keep-around WriteRef failures, so the failed writes from incident 3 were invisible on the Rails side)