Fix concurrent-leave race leaving pool repositories orphaned
What does this MR do and why?
Background
On GitLab.com, orphaned pool_repositories rows have accumulated: pool
records with zero member projects that are stuck in ready state and
never cleaned up. For these rows the Rails database and Gitaly disagree
about what exists — the pool record (and often the object pool on disk)
outlives every project that ever used it. Nothing re-checks pool membership
after a member leaves, so once the cleanup is skipped, the row is orphaned
forever.
One confirmed producer of these orphans is a concurrent-leave race: when the
last two members of a pool leave at the same time, each checks membership
before clearing its own pool_repository_id. Both observe the other as a
remaining member, so both skip mark_obsolete, and the pool is never
scheduled for destruction.
What this MR fixes
This MR closes the concurrent-leave race in Project#leave_pool_repository:
- The Gitaly
disconnect_alternatesRPC runs first, outside any lock (unchanged invariant: a failed disconnect leaves the project a DB member). - Then, inside a
pool.with_lockcritical section, the project clears itspool_repository_idfirst and only then checks for remaining members, callingmark_obsoletewhen none remain.
Clear-then-check under a pool row lock serializes concurrent leavers, so the
last leaver always observes an empty pool and marks it obsolete. The lock is
required (not just the reorder) because Projects::UnlinkForkService calls
this method inside a Project.transaction, where the membership clear is
invisible to concurrent sessions until the outer commit.
All callers get the fix with no signature change: Projects::DestroyService,
Projects::UnlinkForkService, and Repositories::LeavePoolRepositoryWorker.
Part of a larger fix
This MR is one part of the orphaned-pool cleanup effort and fixes only the concurrent-leave producer. The remaining parts are handled as follow-ups:
- Zero-member guard in
ObjectPool::DestroyWorker(closes the window where a fork joins a pool betweenmark_obsoleteand destruction). - Reconciliation sweep to clean up the existing backlog of orphaned rows.
- The storage-move (swap) path race in
Project#swap_pool_repository!.
Note: this branch is based on
!254183 (merged) and includes
its commit; it will be rebased on master once that MR merges.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/628444
- Epic: &19130
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.