Track deletion failure count and alert on stuck project/group deletion

What does this MR do and why?

This is a proposal MR to support the design discussion in #621644 (closed) — it is not a final agreed design. The retry threshold value (currently 5) and admin visibility approach are still open questions in the issue thread.

Background: INC-13111 showed that a project/group stuck in pending_delete because destroy attempts kept silently failing looked identical to a normally-queued deletion for 12 days. Nothing distinguished "failed and retrying forever" from "progressing normally".

This MR implements failure/stuck-detection for entity (project/group) deletion per the scope agreed in the issue discussion:

Changes

1. Track failure count in existing metadata (no new state machine state)

Adds deletion_attempt_count (integer) and deletion_last_failed_at (datetime) to the namespace_detail.state_metadata JSONB column. These are incremented only when reschedule_deletion! is called with a non-blank deletion_error — i.e., only on actual failed destroy attempts, not during normal adjournment/retention waits.

2. Threshold constant with a comment

Projects::DestroyService::MAX_DESTROY_ATTEMPTS = 5 and Groups::DestroyService::MAX_DESTROY_ATTEMPTS = 5 (provisional — the exact number is still under discussion in the issue, see #621644 (comment 3750657992)).

3. Sentry alerting when threshold is crossed

When deletion_attempt_count >= MAX_DESTROY_ATTEMPTS, Gitlab::ErrorTracking.track_exception is called with project/group context. This is the "highest-value, lowest-effort" part called out by the reviewer. Additionally, every failure is tracked in Sentry with deletion_attempt_count in context so the retry count is visible in Sentry/Kibana.

4. sidekiq_retries_exhausted hooks

Added to ProjectDestroyWorker and GroupDestroyWorker to report to Sentry when all Sidekiq retries are exhausted (separate from the attempt-count threshold).

Note: Per the design discussion at #621644 (comment 3773351885), no user-facing messaging is included in this MR. Monitoring is via Sentry (per-failure tracking with deletion_attempt_count in context) and Kibana dashboards.

What is NOT in this MR (per scope agreement)

• No new state machine states
• No new admin/support UI or dashboard (tracked as a follow-up in the issue)
• No broader refactor of the deletion/destroy state machine
• No user-facing stuck-deletion message (per reviewer decision at #621644 (comment 3773351885))

Open questions (from the issue thread)

• Exact threshold value (3 vs 5 vs 10-20)
• Whether to add a Slack channel routing pattern for the Sentry alert
• Admin visibility approach for stuck entities

References

• Issue: #621644 (closed)
• INC-13111 (the incident that motivated this)
• Reviewer design comment: #621644 (comment 3750657992)
• Decision to drop user-facing message: #621644 (comment 3773351885)

How to set up and validate locally

1. In rails console, create a project and simulate repeated destroy failures:

   project = Project.find(<id>)
   user = User.find(<user_id>)
   # Simulate a failed destroy attempt
   project.project_namespace.reschedule_deletion!(transition_user: user, deletion_error: 'Test error')
   project.reload.deletion_attempt_count  # => 1

2. Repeat 4 more times to cross the threshold (5).
3. Check Sentry — a track_exception call should appear with deletion_attempt_count in context.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.