Add loose foreign keys cleanup backlog database diagnostic

What does this MR do and why?

LooseForeignKeys::CleanupWorker drains loose_foreign_keys_deleted_records (and the sibling sharding-key tables) on a 1-minute cron. When it cannot keep up — for example a high-fan-out parent whose child cleanup repeatedly raises a statement timeout — the per-parent-table backlog grows silently. An old backlog can eventually break migrations that assume the orphaned child rows were already cleaned up. This has caused a real incident where public.projects had ~49k pending records with the oldest dating back ~1.5 years, undetected.

This MR adds the backend for a new Loose foreign keys cleanup backlog diagnostic:

  • Gitlab::Database::LooseForeignKeysBacklogChecker reports, per gitlab_shared connection and parent table: pending count, oldest-pending age, and deferred count (consume_after > now()).
  • An on-demand Database::LooseForeignKeysBacklogCheckerWorker (mirrors Database::SchemaCheckerWorker) caches the result.
  • Admin::DatabaseDiagnosticsController endpoints to trigger and read it.

The frontend (admin page) and the Prometheus metrics + automatic cron scheduling follow in separate MRs.

Part of #606252

Queries

The checker runs, per record-store model, one index-only aggregate for the count + deferred-count, plus a single-row lookup per backlogged table for the oldest age. All five tables (loose_foreign_keys_deleted_records and the four loose_foreign_keys_{namespace,organization,project,user}_deleted_records siblings) are structurally identical and each has the same partial index ... (partition, fully_qualified_table_name, consume_after, id) WHERE (status = 1), so the plan shape is the same for all five.

The count aggregate is bounded with a LIMIT cap (PENDING_RECORDS_LIMIT = 10_000) over an index-ordered subquery — no disabled statement timeout (which is a no-op behind PgBouncer anyway). It selects only index columns (fully_qualified_table_name, consume_after), so it is an Index Only Scan that never touches the heap. We scan one row past the cap (LIMIT 10001) so a table that reaches the cap is reported with capped: true and a count clamped to 10,000 (rendered "10,000+").

SELECT fully_qualified_table_name,
       COUNT(*) AS pending_records,
       COUNT(*) FILTER (WHERE consume_after > now()) AS deferred_records
FROM (
  SELECT fully_qualified_table_name, consume_after
  FROM loose_foreign_keys_deleted_records
  WHERE status = 1
  ORDER BY partition, fully_qualified_table_name, consume_after, id
  LIMIT 10001
) sub
GROUP BY fully_qualified_table_name;

The oldest-pending age is read separately as a single indexed row per backlogged table (the head of the cleanup queue), rather than a MIN(created_at) that would scan the whole backlog:

SELECT created_at FROM loose_foreign_keys_deleted_records
WHERE status = 1 AND fully_qualified_table_name = $1
ORDER BY partition, consume_after, id LIMIT 1;

Production results for query 1: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54083/commands/156490

Time: 7.099 ms  
  - planning: 1.164 ms  
  - execution: 5.935 ms  
    - I/O read: 0.503 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 1573 (~12.30 MiB) from the buffer pool  
  - reads: 1 (~8.00 KiB) from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  

Production results for query 2: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54083/commands/156491

Time: 1.659 ms  
  - planning: 1.184 ms  
  - execution: 0.475 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 106 (~848.00 KiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  

Screenshots or screen recordings

N/A (backend only; UI lands in the follow-up frontend MR).

How to set up and validate locally

  1. Seed a backlog on the main connection:
    Gitlab::Database::SharedModel.using_connection(ApplicationRecord.connection) do
      LooseForeignKeys::DeletedRecord.create!(
        fully_qualified_table_name: 'public.projects', primary_key_value: 1, created_at: 3.hours.ago
      )
    end
  2. Gitlab::Database::LooseForeignKeysBacklogChecker.run returns the per-connection backlog.
  3. Database::LooseForeignKeysBacklogCheckerWorker.new.perform, then read Rails.cache.read(Database::LooseForeignKeysBacklogCheckerWorker::BACKLOG_CHECK_CACHE_KEY).

MR acceptance checklist

  • Raw SQL and query plans documented above; db:gitlabcom-database-testing triggered.
  • Tested in all supported migration paths.
  • Reviewed by a database maintainer (new ActiveRecord aggregate scope over a partitioned table).
Edited by Stan Hu

Merge request reports

Loading