Create background migration for db/post_migrate/20170526190000_migrate_build_stage_reference_again.rb
For this migration we should create a background migration and include this in 9.4.0. The steps are as follows (from the top of my head):
In a 9.4.0 post-deployment migration:
- Iterate over all rows in
ci_builds
wherestage_id IS NULL
, selecting onlyid
, in batches of 10 000 - For every id in the batch we'll schedule a background migration job
- Every new batch will be scheduled 5 minutes after the previous batch was scheduled
This ensures there's a 5 minute gap between every batch, giving the database some time to breathe. Based on the number of rows this would take about 7 days to migrate all data. We can adjust the batch sizes and time between batches, they're just example values. The point is that we don't schedule 20 million jobs all at once.