Rewrite LFK cleanup IN query as LATERAL join

What does this MR do?

Rewrites the subquery that LooseForeignKeys::CleanerService uses to identify child rows to delete/update so that it forces one index seek per parent ID via CROSS JOIN LATERAL, instead of generating a single large IN (...) predicate that the planner can mis-cost into a sequential scan.

The new behavior is gated behind the :loose_foreign_keys_lateral_query gitlab_com_derisk feature flag so we can roll it out gradually using Feature.current_request as the actor.

Related: #577552 (comment 3297431559)

Why

When BatchCleanerService calls CleanerService with ~500 parent IDs against a child table whose foreign-key column has moderate cardinality (values not in the most-common-values list, but common enough that the planner over-estimates how many rows match), PostgreSQL can pick a sequential scan over an index scan. We've seen this degrade real production queries to multi-second seq scans that return zero rows — see the plan in https://gitlab.com/gitlab-com/request-for-help/-/issues/3588#note_2826467364.

MATERIALIZED on a CTE doesn't help because the seq scan is on the subquery itself. The structural fix is to evaluate the inner SELECT once per parent ID via a LATERAL join, which forces an index seek per ID. The outer LIMIT preserves the original cap on total rows processed; an inner LIMIT bounds per-parent work.

Query forms

Before: IN (...) subquery

DELETE FROM "issues"
WHERE ("issues"."id") IN (
  SELECT "issues"."id" FROM "issues"
  WHERE "issues"."project_id" IN (1, 2, 3, ...)
  LIMIT 1000
)

example plan

After: CROSS JOIN LATERAL

DELETE FROM "issues"
WHERE ("issues"."id") IN (
  SELECT "lateral_rows"."id"
  FROM unnest(ARRAY[1, 2, 3, ...]) AS parent("project_id")
  CROSS JOIN LATERAL (
    SELECT "issues"."id" FROM "issues"
    WHERE "issues"."project_id" = "parent"."project_id"
    LIMIT 1000
  ) lateral_rows
  LIMIT 1000
)

example plan

Feature flag

Name loose_foreign_keys_lateral_query
Type gitlab_com_derisk
Default false
Actor Feature.current_request

Rollout plan: enable at low percentages on GitLab.com, monitor, then ramp up.

Edited by Krasimir Angelov

Merge request reports

Loading