Back-fill sharding key for events
What does this MR do and why?
This MR back-fills one of the sharding key columns for events:
- Iterate all records without
project_id- This is in order to reduce background migration duration, as records that have
project_idalready have sharding key value. In order to make the planner use the existing index"index_events_on_project_id_and_id" btree (project_id, id)a small change toEachBatchandPrimaryKeyBatchingStrategywas required.
- This is in order to reduce background migration duration, as records that have
- Try to back-fill
project_id/group_idfrom the target model record. - For the records that still have no value for
project_idorgroup_id, try to back-fillpersonal_namespace_idusing the author's personal namespace. - Delete records that still have no sharding key (none of
project_id,group_id, orpersonal_namespace_id).
This MR also modifies EachBatch to allow preset order to be preserved so that we can hint the planner to use the index we need. This is in order to iterate only records without project_id, which greatly reduces the expected duration of the migration.
All possible target models that may have many events were fetched with the following snippet.
[1] pry(main)> Rails.application.eager_load!;
[2] pry(main)> all_models = ActiveRecord::Base.descendants.flat_map {|m| m.descendants};
[3] pry(main)> ap all_models.select {|m| m.reflections.find {|_, r| !r.is_a?(ActiveRecord::Reflection::ThroughReflection) && r.class_name == 'Event' && r.foreign_key == 'target_id'}}.map(&:to_s).uniq.sort
[
[ 0] "DesignManagement::Design",
[ 1] "DiffNote",
[ 2] "DiscussionNote",
[ 3] "Epic",
[ 4] "Issue",
[ 5] "IterationNote",
[ 6] "LabelNote",
[ 7] "LegacyDiffNote",
[ 8] "MergeRequest",
[ 9] "Milestone",
[10] "MilestoneNote",
[11] "Note",
[12] "StateNote",
[13] "SyntheticNote",
[14] "WeightNote",
[15] "WikiPage::Meta",
[16] "WorkItem"
]
=> nil
data-deletion Records left without any sharding key will be deleted. Discussion on this here - !166563 (comment 2162572395).
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
| Before | After |
|---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Related to #462801 (closed)