Track ObjectPool::DestroyWorker retries exhaustion

Emma Parkrequested to merge
616942/destroy-worker-retries-exhausted-logging into master

What does this MR do and why?

GitLab.com has thousands of orphaned pool_repositories records — rows where the Rails DB and Gitaly disagree about what exists (analysis). One root cause is ObjectPool::DestroyWorker failing through all its retries: the job lands in the Sidekiq dead set and the record silently remains in obsolete state as a permanent orphan. Nothing surfaces this today — the 112 currently stuck pools were only discovered months later by comparing DB and Gitaly state (cleanup issue).

As part of fixing that root cause (issue), this MR adds a sidekiq_retries_exhausted hook that tracks the exception with the pool_repository_id, so permanently stuck pools are visible in Sentry when they happen instead of accumulating silently.

Companion to !254157 (merged), which makes the worker idempotent with the default 25 retries — after that lands, this hook only fires for genuinely stuck pools rather than transient hiccups.

References

How to set up and validate locally

  1. In a rails console, simulate retries exhaustion:

    job = { 'args' => [PoolRepository.last.id] }
ObjectPool::DestroyWorker.sidekiq_retries_exhausted_block.call(job, StandardError.new('gitaly failure'))

  2. Verify the exception is tracked (with Sentry disabled locally, it is logged to log/exceptions_json.log including pool_repository_id).

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Emma Park

Merge request reports