## Summary When a reference transaction gets canceled while it is in the prepare phase, the git child is killed while it still holds ref lockfiles (`packed-refs.lock` or `refs/**/*.lock`). Those lockfiles stay on disk. From that point on every `ReplicateRepository` on the affected replica fails with `cannot lock ref ... File exists` (exit status 253) and the replica is stuck on its old generation. The only thing that removes such locks is `CleanStaleData` with its 1h `ReferenceLockfileGracePeriod`, and that only runs as part of housekeeping. A replica whose only remaining traffic is the failing replication retry loop never triggers housekeeping, so the lock stays forever. In our latest incident nothing had cleaned it up almost 3 hours in and the generation gap was still growing. We had to delete the lockfiles by hand. We run a 3-node Gitaly Cluster and have hit this three times in two months, each time through a different RPC (`DeleteRefs`, merge train ref updates, `WriteRef`). The trigger differs but the failure afterwards is always the same. ## Environment - GitLab EE Omnibus 19.2.1 (first two incidents on 19.0.x), git 2.55.0-rc1 - Praefect with 3 Gitaly storages, replication factor 3 - Affected repos are the busy ones (a GitOps repo with frequent merges, merge trains, lots of pipelines) ## Incident 1 (2026-06-05): DeleteRefs A cron-fanned batch of `MergeRequestCleanupRefsWorker` jobs ran at least 5 concurrent `/gitaly.RefService/DeleteRefs` against the same repo. The losers waited 10s on `packed-refs.lock` (`command.real_time_ms` matches `core.packedRefsTimeout`), then died with `unable to prepare: signal: killed: context canceled` while holding the lock. That part is the known #5368 pattern. gitlab-org/gitlab!125109 serializes per merge request, so it does not help against cleanup jobs for different MRs of the same repo. We put a Gitaly concurrency limit on `DeleteRefs` (`max_per_repo: 1`) and that herd has not come back since. ## Incident 2 (2026-06-30): merge train An aborted merge train left locks on `refs/merge-requests/N/{head,train,merge,rebase_on_merge}.lock` plus `packed-refs.lock`, all with the same mtime. Same frozen replication afterwards. ## Incident 3 (2026-08-05, 19.2.1): concurrent WriteRef on a keep-around ref One merge event fanned out 4 `MergeRequests::KeepAroundRefsWorker` jobs within 570ms (same correlation ID). Each writes the same `refs/keep-around/<sha>` via `/gitaly.RepositoryService/WriteRef`. The duplicates collided in voting, Praefect canceled the transaction and update-ref was killed holding the loose ref lock: ``` 07:26:50.710 WriteRef grpc.code=Aborted error="The operation could not be completed. Please try again." error_metadata: reference is locked already, reference=refs/keep-around/<sha> 07:26:50.743 ReferenceTransactionHook error="vote failed: rpc error: code = Canceled desc = transaction has been canceled" transaction.id=4226 07:26:50.753 WriteRef grpc.code=Canceled error="error when running update-ref command: signal: killed: context canceled" ``` `refs/keep-around/<sha>.lock` was left behind on both secondaries. The primary committed and moved on. Note there is no 10s contention timeout involved here, the cancel came about 1s in. Any canceled prepare can do this. From 07:26:57 replication failed every few seconds on both secondaries: ``` ReplicateRepository ... replicating repository: synchronizing references: fetch internal remote: exit status 253 stderr: cannot lock ref 'refs/keep-around/<sha>': Unable to create '.../refs/keep-around/<sha>.lock': File exists ``` `praefect metadata` about 3 hours later, gap still growing: ``` Generation: 372914 (primary) - storage-1: Generation: 372738, behind by 176 changes, Healthy: true, Valid Primary: false - storage-2: Generation: 372738, behind by 176 changes, Healthy: true, Valid Primary: false - storage-3: Generation: 372914, fully up to date, Healthy: true, Valid Primary: true ``` ## Why it never self-heals In incident 1 the locks were reaped 60m21s after creation, which is exactly `ReferenceLockfileGracePeriod`, because housekeeping happened to run on that repo. In incident 3 the gitaly log shows the last scheduled housekeeping for the repo at 05:17 and not a single `removed files` event until we cleaned up manually at 10:13. The lock was born 07:26. So for 2h47m nothing even tried. That makes sense once you see the dependency: `CleanStaleData` runs via housekeeping, housekeeping is triggered by repository activity, and the only activity left on a frozen replica is the replication that the lock itself blocks. The cleanup depends on the operation it is supposed to unblock. ## Impact - Replica frozen for hours until an admin deletes the lockfiles. After that replication heals within a minute. - The frozen replicas keep reporting `Healthy: true`. During the earlier incidents users saw stale reads: freshly pushed branches 404, MRs showing "no changes" and getting auto-closed. The read distribution docs say outdated replicas are excluded from reads. That did not hold up here, or something else routed those reads. Happy to dig out more evidence if that part is interesting. ## Workarounds we use - `max_per_repo: 1` on `DeleteRefs`. Covers incident 1 only, incident 3 came through `WriteRef` anyway. We may evaluate the same limit for `WriteRef`. - As a one-off recovery, deleting the `*.lock` files with an old mtime on the stuck replica works, replication recovers on the next retry. That is how we resolved the incidents so far. ## Suggestions 1. Clean up ref lockfiles when a reference transaction is canceled or aborted, also on the non-WAL path. #5595 punts this to "once transactions are on by default", but transactions do not run under Praefect at all right now, so cluster users never reach that fix. 2. Or run the ref-lock part of `CleanStaleData` when `ReplicateRepository` keeps failing with `cannot lock ref`. The replication worker already knows the exact ref that is stuck. 3. Make `ReferenceLockfileGracePeriod` configurable. 4. On the Praefect side, a replica that is behind and has been failing replication for X minutes should not count as plain healthy. And the read routing exclusion for outdated replicas deserves a check for this state. ## Related - #5595 (killed commands leave temp files behind, blocked on WAL by default, which cluster users cannot run) - #5368 / gitlab-org/gitlab!125109 (DeleteRefs deadlock, fix only serializes within one MR) - #6470 (same "packed-refs locked" error, only proposes a status code change) - !6238 (removes stale packed-refs locks, but only at TransactionManager startup on the WAL path) - gitlab-org/gitlab#591291 (Rails swallows keep-around WriteRef failures, so the failed writes from incident 3 were invisible on the Rails side)