Amend BackfillMergeRequestDiffCommitsToPartitioned
Fix batched background migration BackfillMergeRequestDiffCommitsToPartitioned.
Two problems are addressed:
1. NULL column violations failing entire sub-batches
Rows with NULL commit_author_id, committer_id, or sha in the source table caused NOT NULL violations when inserting into merge_request_commits_metadata (see comment). Because each sub-batch runs as a single SQL statement, one bad row failed the entire sub-batch.
2. Full table scans on GitLab.com due to view batching
On GitLab.com the migration batched over SQL views, but PostgreSQL cannot collapse row-style (multi-column) index conditions through a view, causing full table scans that made batching very slow (see comment).
Changes
- The
filtered_diff_commitsCTE now excludes rows with NULLcommit_author_id,committer_id, orshabefore either INSERT runs. - Both the batching strategy and
sub_batch_relationnow querymerge_request_diff_commitsdirectly instead of the views; the cursor bounds stored on each BBM record continue to isolate the four parallel workers by range. - A fix migration (
FixBackfillMergeRequestDiffCommitsToPartitioned) updates existing BBMs in place:sub_batch_size1000 → 10000 andbatch_size50000 → 500000 to take advantage of the faster queries. total_tuple_countis also backfilled for GitLab.com since views have nopg_classcardinality (based on this pattern).- The original queue migration is left unchanged since this fix ships in the same milestone, and the migration has not yet started on self-managed instances. On GitLab.com, failed jobs will be retried with the new code before the migration finalizes.
References
Related to #527230+
Original MR: Backfill partitioned `merge_request_diff_commit... (!224072 - merged)
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.