[19.0] Rewrite LFK cleanup query as LATERAL join

What does this MR do and why?

Backports the Loose Foreign Keys (LFK) cleanup query performance fix to the 19-0-stable-ee branch, for self-managed and GitLab Dedicated instances running 19.0.x.

It cherry-picks the GA (post-flag-removal) form so the fix is on by default, since the feature flag no longer exists in later releases:

1. !235721 (merged) — Rewrite LFK cleanup IN query as a LATERAL join (e9d8803ca00e)
2. !240768 (merged) — Remove the loose_foreign_keys_lateral_query feature flag (e7fe4677c591)

Why

On large instances LooseForeignKeys::CleanupWorker falls behind because the cleanup queries time out. The pre-19.2 query form is a correlated IN (SELECT ... WHERE fk_col IN ($1...$500) LIMIT N FOR UPDATE SKIP LOCKED) against high-fan-out parents, which on partitioned children (ci_builds, p_ci_pipelines) hits PG::QueryCanceled: canceling statement due to statement timeout. When these time out, the parent record is never marked processed, loose_foreign_keys_deleted_records grows unbounded, and old partitions never drop.

The LATERAL join form forces one index seek per parent id instead of a single large scan. PartitionCleanerService inherits it, so partitioned children benefit too.

This directly caused the upgrade incident in https://gitlab.com/gitlab-org/gitlab/-/issues/605940 (a packages_helm_metadata_cache_states sharding-key backfill hit a projects FK violation because a deleted project's cleanup record had been stuck pending for months).

Conflict resolution

Only spec/services/loose_foreign_keys/cleaner_service_spec.rb (FOSS) conflicted, because an intermediate rubocop commit (692fb3788073 — "Fix RSpec/BeEq Rubocop offenses") on master is not on the stable branch. Resolved by taking the end-state of that spec file from e7fe4677c591; the resulting cleaner_service.rb and specs are identical to the master post-flag-removal state.

References

• Backport tracking issue: #606254 (closed)
• Root-cause / failure-mode collection: #600628
• Upgrade incident: https://gitlab.com/gitlab-org/gitlab/-/issues/605940
• Backlog monitoring follow-up: #606252

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
• The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
• The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
• Required labels have been applied to this merge request
  • severity label and bug subtype labels (if applicable)
  • If this MR fixes a bug that affects customers, the customer label has been applied.
• This MR has been approved by a maintainer (only one approval is required).
• Ensure the e2e:test-on-omnibus-ee job has succeeded, or if it has failed, investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

• Refer to the patch release runbook for engineers and maintainers for guidance.
• Ask questions on the #releases Slack channel (internal only).
• Once the backport has been merged, the commit changes will be automatically deployed to a release environment that can be used for manual validation. See after merging runbook for details.