Back-fill sharding key for events

What does this MR do and why?

This MR back-fills one of the sharding key columns for events:

  • Iterate all records without project_id
    • This is in order to reduce background migration duration, as records that have project_id already have sharding key value. In order to make the planner use the existing index "index_events_on_project_id_and_id" btree (project_id, id) a small change to EachBatch and PrimaryKeyBatchingStrategy was required.
  • Try to back-fill project_id / group_id from the target model record.
  • For the records that still have no value for project_id or group_id, try to back-fill personal_namespace_id using the author's personal namespace.
  • Delete records that still have no sharding key (none of project_id, group_id, or personal_namespace_id).

This MR also modifies EachBatch to allow preset order to be preserved so that we can hint the planner to use the index we need. This is in order to iterate only records without project_id, which greatly reduces the expected duration of the migration.

All possible target models that may have many events were fetched with the following snippet.

[1] pry(main)> Rails.application.eager_load!;
[2] pry(main)> all_models = ActiveRecord::Base.descendants.flat_map {|m| m.descendants};
[3] pry(main)> ap all_models.select {|m| m.reflections.find {|_, r| !r.is_a?(ActiveRecord::Reflection::ThroughReflection) && r.class_name == 'Event' && r.foreign_key == 'target_id'}}.map(&:to_s).uniq.sort
[
    [ 0] "DesignManagement::Design",
    [ 1] "DiffNote",
    [ 2] "DiscussionNote",
    [ 3] "Epic",
    [ 4] "Issue",
    [ 5] "IterationNote",
    [ 6] "LabelNote",
    [ 7] "LegacyDiffNote",
    [ 8] "MergeRequest",
    [ 9] "Milestone",
    [10] "MilestoneNote",
    [11] "Note",
    [12] "StateNote",
    [13] "SyntheticNote",
    [14] "WeightNote",
    [15] "WikiPage::Meta",
    [16] "WorkItem"
]
=> nil

data-deletion Records left without any sharding key will be deleted. Discussion on this here - !166563 (comment 2162572395).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Related to #462801 (closed)

Edited by Krasimir Angelov

Merge request reports

Loading