Apply B-tree folding fix to `BackfillMergeRequestDiffCommitsToPartitioned`
What does this MR do and why?
Update BackfillMergeRequestDiffCommitsToPartitioned#sub_batch_relation override to use InclusiveCursorIterator introduced in !235758 (merged) (re #599681 (closed))
Also, raise max_batch_size from its original conservative cap to 2_000_000 on Gitlab.com and set the correct value to desired_sharding_key_migration_job_name in the migrated table dictionary file (re #581114 (closed))
References
Related to #527230
DB
Each sub-batch now has exactly one lower bound and one upper bound on the cursor columns. With a single pair, PostgreSQL can translate both directly into B-tree index range bounds. Before, two competing lower bounds (>= and >) forced the planner to pick one as the index range and apply the other as a row filter — so every sub-batch re-scanned from start_cursor regardless of where it actually needed to start.
Before
Sub-batch iterator query (every sub-batch, all iterations):
SELECT "merge_request_diff_commits".*
FROM "merge_request_diff_commits"
WHERE ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") >= ($1, $2) -- start_cursor, from scope
AND ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") <= ($3, $4) -- end_cursor, from scope
AND (("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") > ($5, $6)) -- prev_end, from Iterator
ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
"merge_request_diff_commits"."relative_order" ASC
LIMIT $8
OFFSET $7Full CTE (same pattern inside sub_batch):
WITH sub_batch AS MATERIALIZED (
SELECT "merge_request_diff_commits".*
FROM "merge_request_diff_commits"
WHERE ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") >= ($1, $2)
AND ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") <= ($3, $4)
AND (("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") > ($5, $6))
ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
"merge_request_diff_commits"."relative_order" ASC
LIMIT $7
),
diff_commits AS MATERIALIZED ( ... ),
...After
Phase 1 - first sub-batch (InclusiveCursorIterator: >= start_cursor)
The scope carries only the upper bound; inclusive_start_predicate adds >= for this batch only:
SELECT "merge_request_diff_commits".*
FROM "merge_request_diff_commits"
WHERE ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") <= ($1, $2) -- end_cursor, from scope
AND ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") >= ($3, $4) -- start_cursor, from InclusiveCursorIterator
ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
"merge_request_diff_commits"."relative_order" ASC
LIMIT $5Full CTE (phase 1):
WITH sub_batch AS MATERIALIZED (
SELECT "merge_request_diff_commits".*
FROM "merge_request_diff_commits"
WHERE ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") <= ($1, $2)
AND ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") >= ($3, $4)
ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
"merge_request_diff_commits"."relative_order" ASC
LIMIT $5
),
diff_commits AS MATERIALIZED ( ... ),
...Phase 2 - subsequent sub-batches (Iterator: > prev_end)
The scope still carries only the upper bound; the Iterator adds > against its seeded cursor:
SELECT "merge_request_diff_commits".*
FROM "merge_request_diff_commits"
WHERE ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") <= ($1, $2) -- end_cursor, from scope
AND (("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") > ($3, $4)) -- prev_end, from Iterator
ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
"merge_request_diff_commits"."relative_order" ASC
LIMIT $5Full CTE (phase 2):
WITH sub_batch AS MATERIALIZED (
SELECT "merge_request_diff_commits".*
FROM "merge_request_diff_commits"
WHERE ("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") <= ($1, $2)
AND (("merge_request_diff_commits"."merge_request_diff_id",
"merge_request_diff_commits"."relative_order") > ($3, $4))
ORDER BY "merge_request_diff_commits"."merge_request_diff_id" ASC,
"merge_request_diff_commits"."relative_order" ASC
LIMIT $5
),
diff_commits AS MATERIALIZED ( ... ),
...MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.