Phase 2 (b) - DeletedRecordStore facade with feature flag

What does this MR do and why?

Make LFKs compatible with cells phase 2 (b).

This MR introduces Gitlab::LooseForeignKeys::DeletedRecordStore, a stateless facade that fans LFK cleanup operations across the 5 LFK tables. The 3 LFK workers are wired to choose between the legacy model and the facade based on a new use_loose_foreign_keys_deleted_record_store feature flag, disabled by default. With the flag off, production behavior is identical to today. With the flag on, the facade queries the still empty sharded tables + the actual loose_foreign_keys_deleted_records table.

Builds on the record_store: seam introduced in !237757 (merged) (Phase 2 (a)).

Phase MR
1 - Foundation !233234 (merged)
2 (a) - Inject record_store into LFK services !237757 (merged)
2 (b) - DeletedRecordStore facade + workers + feature flag This MR
3 - Trigger function update + first table rollout -
4 - Gradual trigger rollout -
5 - Cleanup -

About the changes

  1. DeletedRecordStore (Facade): Stateless class with class methods only. Routes the methods to each model, which shares the same concern app/models/concerns/loose_foreign_keys/deleted_record_concern.rb
  2. Feature flag: use_loose_foreign_keys_deleted_record_store, default_enabled: false. Will be enabled per-worker in production via /chatops run feature set use_loose_foreign_keys_deleted_record_store true after deploy.
  3. Workers: CleanupWorker, CiPipelinesBuildsCleanupCronWorker, and MergeRequestDiffCommitCleanupWorker each pass record_store: record_store<DeletedRecordStore> into ProcessDeletedRecordsService.new.

Empty-table safety

With the flag on, load_batch_for_table issues 5 queries instead of 1; 4 return zero rows until Phase 3 starts routing inserts via the trigger function. The query count is bounded and explicit. Phase 1's migration already created the initial partition per new table, so there is nothing to set up at runtime.

Steady-state memory profile

load_batch_for_table queries every model with the full batch_size and merge-sorts the results to preserve the global [partition_number, consume_after, id] consume order, then truncates to batch_size. Worst case is MODELS.size * batch_size records loaded in Ruby per call (≈ 5 × 500 = 2,500 small AR objects per cycle, a few MB).

Until Phase 3 routes inserts via the trigger function, four of the five siblings return 0 rows, so steady-state load is just batch_size. The intent and tradeoffs are documented inline on Gitlab::LooseForeignKeys::DeletedRecordStore.load_batch_for_table. Alternative strategies (per-model cap, hybrid cap + redistribute) are tracked in #597949 and can be revisited once Phase 3 monitoring shows real distribution across the sibling tables.

Observability

Per-model rollout metrics are emitted by Gitlab::Metrics::LooseForeignKeysDeletedRecordStore. They are short-lived facade observability and will be removed in Phase 5 alongside the feature flag. The existing long-term counters in LooseForeignKeys::BatchCleanerService (loose_foreign_key_processed_deleted_records, loose_foreign_key_rescheduled_deleted_records, loose_foreign_key_incremented_deleted_records) are untouched, so current dashboards and alerts are unaffected.

Related #597949

Edited by Leonardo da Rosa

Merge request reports

Loading