Fix concurrent-leave race leaving pool repositories orphaned

What does this MR do and why?

Background

On GitLab.com, orphaned pool_repositories rows have accumulated: pool records with zero member projects that are stuck in ready state and never cleaned up. For these rows the Rails database and Gitaly disagree about what exists — the pool record (and often the object pool on disk) outlives every project that ever used it. Nothing re-checks pool membership after a member leaves, so once the cleanup is skipped, the row is orphaned forever.

One confirmed producer of these orphans is a concurrent-leave race: when the last two members of a pool leave at the same time, each checks membership before clearing its own pool_repository_id. Both observe the other as a remaining member, so both skip mark_obsolete, and the pool is never scheduled for destruction.

What this MR fixes

This MR closes the concurrent-leave race in Project#leave_pool_repository:

  1. The Gitaly disconnect_alternates RPC runs first, outside any lock (unchanged invariant: a failed disconnect leaves the project a DB member).
  2. Then, inside a pool.with_lock critical section, the project clears its pool_repository_id first and only then checks for remaining members, calling mark_obsolete when none remain.

Clear-then-check under a pool row lock serializes concurrent leavers, so the last leaver always observes an empty pool and marks it obsolete. The lock is required (not just the reorder) because Projects::UnlinkForkService calls this method inside a Project.transaction, where the membership clear is invisible to concurrent sessions until the outer commit.

All callers get the fix with no signature change: Projects::DestroyService, Projects::UnlinkForkService, and Repositories::LeavePoolRepositoryWorker.

Part of a larger fix

This MR is one part of the orphaned-pool cleanup effort and fixes only the concurrent-leave producer. The remaining parts are handled as follow-ups:

  • Zero-member guard in ObjectPool::DestroyWorker (closes the window where a fork joins a pool between mark_obsolete and destruction).
  • Reconciliation sweep to clean up the existing backlog of orphaned rows.
  • The storage-move (swap) path race in Project#swap_pool_repository!.

Note: this branch is based on !254183 (merged) and includes its commit; it will be rebased on master once that MR merges.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Emma Park

Merge request reports

Loading
Loading