Add loose foreign keys cleanup backlog database diagnostic
What does this MR do and why?
LooseForeignKeys::CleanupWorker drains loose_foreign_keys_deleted_records
(and the sibling sharding-key tables) on a 1-minute cron. When it cannot keep
up — for example a high-fan-out parent whose child cleanup repeatedly raises a
statement timeout — the per-parent-table backlog grows silently. An old backlog
can eventually break migrations that assume the orphaned child rows were already
cleaned up. This has caused a real incident where public.projects had ~49k
pending records with the oldest dating back ~1.5 years, undetected.
This MR adds the backend for a new Loose foreign keys cleanup backlog diagnostic:
Gitlab::Database::LooseForeignKeysBacklogCheckerreports, per gitlab_shared connection and parent table: pending count, oldest-pending age, and deferred count (consume_after > now()).- An on-demand
Database::LooseForeignKeysBacklogCheckerWorker(mirrorsDatabase::SchemaCheckerWorker) caches the result. Admin::DatabaseDiagnosticsControllerendpoints to trigger and read it.
The frontend (admin page) and the Prometheus metrics + automatic cron scheduling follow in separate MRs.
Part of #606252
Queries
The checker runs, per record-store model, one index-only aggregate for the count +
deferred-count, plus a single-row lookup per backlogged table for the oldest age. All five
tables (loose_foreign_keys_deleted_records and the four
loose_foreign_keys_{namespace,organization,project,user}_deleted_records siblings) are
structurally identical and each has the same partial index
... (partition, fully_qualified_table_name, consume_after, id) WHERE (status = 1), so the
plan shape is the same for all five.
The count aggregate is bounded with a LIMIT cap (PENDING_RECORDS_LIMIT = 10_000) over
an index-ordered subquery — no disabled statement timeout (which is a no-op behind PgBouncer
anyway). It selects only index columns (fully_qualified_table_name, consume_after), so it
is an Index Only Scan that never touches the heap. We scan one row past the cap
(LIMIT 10001) so a table that reaches the cap is reported with capped: true and a count
clamped to 10,000 (rendered "10,000+").
SELECT fully_qualified_table_name,
COUNT(*) AS pending_records,
COUNT(*) FILTER (WHERE consume_after > now()) AS deferred_records
FROM (
SELECT fully_qualified_table_name, consume_after
FROM loose_foreign_keys_deleted_records
WHERE status = 1
ORDER BY partition, fully_qualified_table_name, consume_after, id
LIMIT 10001
) sub
GROUP BY fully_qualified_table_name;The oldest-pending age is read separately as a single indexed row per backlogged table (the
head of the cleanup queue), rather than a MIN(created_at) that would scan the whole backlog:
SELECT created_at FROM loose_foreign_keys_deleted_records
WHERE status = 1 AND fully_qualified_table_name = $1
ORDER BY partition, consume_after, id LIMIT 1;Production results for query 1: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54083/commands/156490
Time: 7.099 ms
- planning: 1.164 ms
- execution: 5.935 ms
- I/O read: 0.503 ms
- I/O write: 0.000 ms
Shared buffers:
- hits: 1573 (~12.30 MiB) from the buffer pool
- reads: 1 (~8.00 KiB) from the OS file cache, including disk I/O
- dirtied: 0
- writes: 0 Production results for query 2: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/54083/commands/156491
Time: 1.659 ms
- planning: 1.184 ms
- execution: 0.475 ms
- I/O read: 0.000 ms
- I/O write: 0.000 ms
Shared buffers:
- hits: 106 (~848.00 KiB) from the buffer pool
- reads: 0 from the OS file cache, including disk I/O
- dirtied: 0
- writes: 0 Screenshots or screen recordings
N/A (backend only; UI lands in the follow-up frontend MR).
How to set up and validate locally
- Seed a backlog on the
mainconnection:Gitlab::Database::SharedModel.using_connection(ApplicationRecord.connection) do LooseForeignKeys::DeletedRecord.create!( fully_qualified_table_name: 'public.projects', primary_key_value: 1, created_at: 3.hours.ago ) end Gitlab::Database::LooseForeignKeysBacklogChecker.runreturns the per-connection backlog.Database::LooseForeignKeysBacklogCheckerWorker.new.perform, then readRails.cache.read(Database::LooseForeignKeysBacklogCheckerWorker::BACKLOG_CHECK_CACHE_KEY).
MR acceptance checklist
- Raw SQL and query plans documented above;
db:gitlabcom-database-testingtriggered. - Tested in all supported migration paths.
- Reviewed by a database maintainer (new ActiveRecord aggregate scope over a partitioned table).