Add pending-delete backlog age metric

What does this MR do and why?

Adds a Prometheus sampler exporting the state of the deletion-scheduled backlog, so a broken deletion pipeline is caught within hours instead of days.

During INC-13111, project and group deletion was fully broken on GitLab.com for 12 days. A record scheduled for deletion but never processed looks identical to one that is simply queued, and we had no detection for it.

Depends on

  • Stacked on !254380 (merged) (index on namespace_details.deletion_scheduled_at). This MR targets that branch, so its diff shows only the metric code. Merge !254380 (merged) first (and confirm the index is live on GitLab.com); this MR will then retarget to master automatically.

Implementation

  • New Gitlab::Metrics::Samplers::PendingDeleteBacklogSampler (Sidekiq, every 5 min, ExclusiveLeaseGuard, reads primary), behind the pending_delete_backlog_metric ops flag (off by default). Exports two gauges:
    • gitlab_deletion_scheduled_backlog_oldest_age_seconds — age of the oldest record awaiting deletion.
    • gitlab_deletion_scheduled_backlog_overdue_count — number of records scheduled longer than deletion_adjourned_period + 1 day.
  • Both are sourced from namespace_details.deletion_scheduled_at, which covers projects and groups (scheduling either sets that column via the shared state machine). Uses Namespace::Detail.oldest_deletion_scheduled_at and the existing deletion_scheduled_before scope.

Review changes applied

  • Reuse over a project-specific query: dropped Project.oldest_pending_delete_marked_at; everything now flows through namespace_details.
  • Skew concern: added the overdue-count gauge so a single stuck record doesn't dominate the signal.
  • Terminology: dropped pending_delete/resource-label wording in favour of the deletion_scheduled state-machine terms.
  • Index-first: the supporting index moved to its own MR (!254380 (merged)) per the migration-type guidance.

Query plans

Both queries hit the new partial index index_namespace_details_on_deletion_scheduled_at (added in !254380 (merged)):

  • SELECT MIN(deletion_scheduled_at) FROM namespace_details WHERE deletion_scheduled_at IS NOT NULL — index-only scan, single row, 0.128 ms on a production-scale Database Lab clone (plan).
  • SELECT COUNT(*) FROM namespace_details WHERE deletion_scheduled_at <= <cutoff> — same partial index; Database Lab plan to be attached.

How to set up and validate locally

  1. Feature.enable(:pending_delete_backlog_metric)
  2. Schedule a project or group for deletion with an old deletion_scheduled_at.
  3. Trigger the sampler; confirm gitlab_deletion_scheduled_backlog_oldest_age_seconds reflects the age and ..._overdue_count counts records past the window.

MR acceptance checklist

  • Index MR !254380 (merged) merged and confirmed live on GitLab.com.
  • Raw SQL + Database Lab plan for the oldest-age query documented.
  • Database Lab plan for the overdue-count query attached.
  • Tests added for enabled/disabled flag states, empty backlog, and overdue counting.

Related to #621640

Edited by Aakriti Gupta

Merge request reports

Loading
Loading