Loading
Add pending-delete backlog age metric
What does this MR do and why?
Adds a Prometheus sampler exporting the state of the deletion-scheduled backlog, so a broken deletion pipeline is caught within hours instead of days.
During INC-13111, project and group deletion was fully broken on GitLab.com for 12 days. A record scheduled for deletion but never processed looks identical to one that is simply queued, and we had no detection for it.
Depends on
- Stacked on !254380 (merged) (index on
namespace_details.deletion_scheduled_at). This MR targets that branch, so its diff shows only the metric code. Merge !254380 (merged) first (and confirm the index is live on GitLab.com); this MR will then retarget tomasterautomatically.
Implementation
- New
Gitlab::Metrics::Samplers::PendingDeleteBacklogSampler(Sidekiq, every 5 min,ExclusiveLeaseGuard, reads primary), behind thepending_delete_backlog_metricops flag (off by default). Exports two gauges:gitlab_deletion_scheduled_backlog_oldest_age_seconds— age of the oldest record awaiting deletion.gitlab_deletion_scheduled_backlog_overdue_count— number of records scheduled longer thandeletion_adjourned_period + 1 day.
- Both are sourced from
namespace_details.deletion_scheduled_at, which covers projects and groups (scheduling either sets that column via the shared state machine). UsesNamespace::Detail.oldest_deletion_scheduled_atand the existingdeletion_scheduled_beforescope.
Review changes applied
- Reuse over a project-specific query: dropped
Project.oldest_pending_delete_marked_at; everything now flows throughnamespace_details. - Skew concern: added the overdue-count gauge so a single stuck record doesn't dominate the signal.
- Terminology: dropped
pending_delete/resource-label wording in favour of thedeletion_scheduledstate-machine terms. - Index-first: the supporting index moved to its own MR (!254380 (merged)) per the migration-type guidance.
Query plans
Both queries hit the new partial index index_namespace_details_on_deletion_scheduled_at (added in !254380 (merged)):
SELECT MIN(deletion_scheduled_at) FROM namespace_details WHERE deletion_scheduled_at IS NOT NULL— index-only scan, single row, 0.128 ms on a production-scale Database Lab clone (plan).SELECT COUNT(*) FROM namespace_details WHERE deletion_scheduled_at <= <cutoff>— same partial index; Database Lab plan to be attached.
How to set up and validate locally
Feature.enable(:pending_delete_backlog_metric)- Schedule a project or group for deletion with an old
deletion_scheduled_at. - Trigger the sampler; confirm
gitlab_deletion_scheduled_backlog_oldest_age_secondsreflects the age and..._overdue_countcounts records past the window.
MR acceptance checklist
- Index MR !254380 (merged) merged and confirmed live on GitLab.com.
- Raw SQL + Database Lab plan for the oldest-age query documented.
- Database Lab plan for the overdue-count query attached.
- Tests added for enabled/disabled flag states, empty backlog, and overdue counting.
Related to #621640
Edited by Aakriti Gupta