Guard object pool destruction against late joiners

What does this MR do and why?

Background

On GitLab.com, orphaned pool_repositories rows have accumulated: pool records with zero member projects that are stuck in ready state and never cleaned up. For these rows the Rails database and Gitaly disagree about what exists — the pool record (and often the object pool on disk) outlives every project that ever used it. Nothing re-checks pool membership after a member leaves, so once the cleanup is skipped, the row is orphaned forever.

A related producer of pool inconsistency is a late-joiner race: Projects::ForkService assigns pool membership to a new fork at creation time without checking the pool's state. A fork can therefore join a pool in the window between the pool being marked obsolete (which schedules ObjectPool::DestroyWorker) and the worker actually running. The worker deleted any obsolete pool without re-checking membership, so it could destroy an object pool that a live fork's repository still depends on via Git alternates.

What this MR fixes

This MR guards ObjectPool::DestroyWorker against late joiners: after the existing obsolete? check in ObjectPool::DestroyWorker#perform, the worker now skips destruction when pool.member_projects.exists?, and logs the skip (destroy_skipped: members_exist) so occurrences show up in Kibana.

A skipped pool self-heals: it stays obsolete with its member, and when that last member later leaves, mark_obsolete fires again and re-enqueues the worker. A spec covers this.

There are no schema changes — the membership check uses projects.pool_repository_id, which is already indexed. The normal flow is unaffected: pools are only marked obsolete with zero members, so the guard only acts in the race.

One residual gap is intentionally out of scope: a fork can still join after the guard's check but before pool.destroy. Fully closing that requires the join path to check pool state under the pool lock, which is a follow-up under the same work item.

Part of a larger fix

This MR is one part of the orphaned-pool cleanup effort and fixes only the late-joiner producer. The concurrent-leave race was already fixed in !254730 (merged). The remaining parts are handled as follow-ups:

  • A reconciliation sweep to clean up the existing backlog of orphaned rows.
  • The storage-move (swap) path race in Project#swap_pool_repository!.

Query Plan

SELECT 1 AS one
FROM "projects"
WHERE "projects"."pool_repository_id" = <id>
LIMIT 1

https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/57435/commands/161934

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Emma Park

Merge request reports

Loading
Loading