Guard object pool destruction against late joiners
What does this MR do and why?
Background
On GitLab.com, orphaned pool_repositories rows have accumulated: pool
records with zero member projects that are stuck in ready state and
never cleaned up. For these rows the Rails database and Gitaly disagree
about what exists — the pool record (and often the object pool on disk)
outlives every project that ever used it. Nothing re-checks pool membership
after a member leaves, so once the cleanup is skipped, the row is orphaned
forever.
A related producer of pool inconsistency is a late-joiner race:
Projects::ForkService assigns pool membership to a new fork at creation
time without checking the pool's state. A fork can therefore join a pool in
the window between the pool being marked obsolete (which schedules
ObjectPool::DestroyWorker) and the worker actually running. The worker
deleted any obsolete pool without re-checking membership, so it could
destroy an object pool that a live fork's repository still depends on via
Git alternates.
What this MR fixes
This MR guards ObjectPool::DestroyWorker against late joiners: after the
existing obsolete? check in ObjectPool::DestroyWorker#perform, the
worker now skips destruction when pool.member_projects.exists?, and logs
the skip (destroy_skipped: members_exist) so occurrences show up in
Kibana.
A skipped pool self-heals: it stays obsolete with its member, and when
that last member later leaves, mark_obsolete fires again and re-enqueues
the worker. A spec covers this.
There are no schema changes — the membership check uses
projects.pool_repository_id, which is already indexed. The normal flow is
unaffected: pools are only marked obsolete with zero members, so the guard
only acts in the race.
One residual gap is intentionally out of scope: a fork can still join after
the guard's check but before pool.destroy. Fully closing that requires the
join path to check pool state under the pool lock, which is a follow-up
under the same work item.
Part of a larger fix
This MR is one part of the orphaned-pool cleanup effort and fixes only the late-joiner producer. The concurrent-leave race was already fixed in !254730 (merged). The remaining parts are handled as follow-ups:
- A reconciliation sweep to clean up the existing backlog of orphaned rows.
- The storage-move (swap) path race in
Project#swap_pool_repository!.
Query Plan
SELECT 1 AS one
FROM "projects"
WHERE "projects"."pool_repository_id" = <id>
LIMIT 1https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/57435/commands/161934
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/628444
- Epic: &19130
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.