Apply B-tree folding fix to `BackfillMergeRequestDiffCommitsToPartitioned`

What does this MR do and why?

Update BackfillMergeRequestDiffCommitsToPartitioned#sub_batch_relation override to use InclusiveCursorIterator introduced in !235758 (merged) (re #599681 (closed))

Also, raise max_batch_size from its original conservative cap to 2_000_000 on Gitlab.com and set the correct value to desired_sharding_key_migration_job_name in the migrated table dictionary file (re #581114 (closed))

References

Related to #527230

DB

Each sub-batch now has exactly one lower bound and one upper bound on the cursor columns. With a single pair, PostgreSQL can translate both directly into B-tree index range bounds. Before, two competing lower bounds (>= and >) forced the planner to pick one as the index range and apply the other as a row filter — so every sub-batch re-scanned from start_cursor regardless of where it actually needed to start.

Before

Sub-batch iterator query (every sub-batch, all iterations):

  SELECT "merge_request_diff_commits".*
  FROM "merge_request_diff_commits"
  WHERE ("merge_request_diff_commits"."merge_request_diff_id",
         "merge_request_diff_commits"."relative_order") >= ($1, $2)   -- start_cursor, from scope
    AND ("merge_request_diff_commits"."merge_request_diff_id",
         "merge_request_diff_commits"."relative_order") <= ($3, $4)   -- end_cursor, from scope
    AND (("merge_request_diff_commits"."merge_request_diff_id",
          "merge_request_diff_commits"."relative_order") > ($5, $6))  -- prev_end, from Iterator
  ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
           "merge_request_diff_commits"."relative_order" ASC
  LIMIT $8
  OFFSET $7

Full CTE (same pattern inside sub_batch):

  WITH sub_batch AS MATERIALIZED (
    SELECT "merge_request_diff_commits".*
    FROM "merge_request_diff_commits"
    WHERE ("merge_request_diff_commits"."merge_request_diff_id",
           "merge_request_diff_commits"."relative_order") >= ($1, $2)
      AND ("merge_request_diff_commits"."merge_request_diff_id",
           "merge_request_diff_commits"."relative_order") <= ($3, $4)
      AND (("merge_request_diff_commits"."merge_request_diff_id",
            "merge_request_diff_commits"."relative_order") > ($5, $6))
    ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
             "merge_request_diff_commits"."relative_order" ASC
    LIMIT $7
  ),
  diff_commits AS MATERIALIZED ( ... ),
  ...

After

Phase 1 - first sub-batch (InclusiveCursorIterator: >= start_cursor)

The scope carries only the upper bound; inclusive_start_predicate adds >= for this batch only:

  SELECT "merge_request_diff_commits".*
  FROM "merge_request_diff_commits"
  WHERE ("merge_request_diff_commits"."merge_request_diff_id",
         "merge_request_diff_commits"."relative_order") <= ($1, $2)   -- end_cursor, from scope
    AND ("merge_request_diff_commits"."merge_request_diff_id",
         "merge_request_diff_commits"."relative_order") >= ($3, $4)   -- start_cursor, from InclusiveCursorIterator
  ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
           "merge_request_diff_commits"."relative_order" ASC
  LIMIT $5

Full CTE (phase 1):

  WITH sub_batch AS MATERIALIZED (
    SELECT "merge_request_diff_commits".*
    FROM "merge_request_diff_commits"
    WHERE ("merge_request_diff_commits"."merge_request_diff_id",
           "merge_request_diff_commits"."relative_order") <= ($1, $2)
      AND ("merge_request_diff_commits"."merge_request_diff_id",
           "merge_request_diff_commits"."relative_order") >= ($3, $4)
    ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
             "merge_request_diff_commits"."relative_order" ASC
    LIMIT $5
  ),
  diff_commits AS MATERIALIZED ( ... ),
  ...

Phase 2 - subsequent sub-batches (Iterator: > prev_end)

The scope still carries only the upper bound; the Iterator adds > against its seeded cursor:

  SELECT "merge_request_diff_commits".*
  FROM "merge_request_diff_commits"
  WHERE ("merge_request_diff_commits"."merge_request_diff_id",
         "merge_request_diff_commits"."relative_order") <= ($1, $2)   -- end_cursor, from scope
    AND (("merge_request_diff_commits"."merge_request_diff_id",
          "merge_request_diff_commits"."relative_order") > ($3, $4))  -- prev_end, from Iterator
  ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
           "merge_request_diff_commits"."relative_order" ASC
  LIMIT $5

Full CTE (phase 2):

  WITH sub_batch AS MATERIALIZED (
    SELECT "merge_request_diff_commits".*
    FROM "merge_request_diff_commits"
    WHERE ("merge_request_diff_commits"."merge_request_diff_id",
           "merge_request_diff_commits"."relative_order") <= ($1, $2)
      AND (("merge_request_diff_commits"."merge_request_diff_id",
            "merge_request_diff_commits"."relative_order") > ($3, $4))
    ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
             "merge_request_diff_commits"."relative_order" ASC
    LIMIT $5
  ),
  diff_commits AS MATERIALIZED ( ... ),
  ...

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eugenia Grieff

Merge request reports

Loading
Loading