Amend BackfillMergeRequestDiffCommitsToPartitioned

Fix batched background migration BackfillMergeRequestDiffCommitsToPartitioned.

Two problems are addressed:

1. NULL column violations failing entire sub-batches

Rows with NULL commit_author_id, committer_id, or sha in the source table caused NOT NULL violations when inserting into merge_request_commits_metadata (see comment). Because each sub-batch runs as a single SQL statement, one bad row failed the entire sub-batch.

2. Full table scans on GitLab.com due to view batching

On GitLab.com the migration batched over SQL views, but PostgreSQL cannot collapse row-style (multi-column) index conditions through a view, causing full table scans that made batching very slow (see comment).

Changes

  • The filtered_diff_commits CTE now excludes rows with NULL commit_author_id, committer_id, or sha before either INSERT runs.
  • Both the batching strategy and sub_batch_relation now query merge_request_diff_commits directly instead of the views; the cursor bounds stored on each BBM record continue to isolate the four parallel workers by range.
  • A fix migration (FixBackfillMergeRequestDiffCommitsToPartitioned) updates existing BBMs in place: sub_batch_size 1000 → 10000 and batch_size 50000 → 500000 to take advantage of the faster queries.
  • total_tuple_count is also backfilled for GitLab.com since views have no pg_class cardinality (based on this pattern).
  • The original queue migration is left unchanged since this fix ships in the same milestone, and the migration has not yet started on self-managed instances. On GitLab.com, failed jobs will be retried with the new code before the migration finalizes.

References

Related to #527230+

Original MR: Backfill partitioned `merge_request_diff_commit... (!224072 - merged)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Eugenia Grieff

Merge request reports

Loading
Loading