Fix BackfillDesiredShardingKeyJob batched migration job.
Problem
BackfillDesiredShardingKeyJob batched migration is a base class use to back-fill sharding key for a table. It's used by Housekeeper to create MRs by consuming the desired sharding key information from the DB dictionary.
The migration has a custom filter like
scope_to ->(relation) { relation.where(backfill_column => nil) }
This is done in order to skip rows that already have sharding key. There are problems with this approach - tables rarely have the required index to support this iteration, which leads to the database filtering on the fly (Rows removed by filter...). In many cases it just works, as none of the rows have the sharding key, so the database is able to quickly find the batch boundaries, but it fails when the migration is rescheduled (example), or when reaches the end of the table where we may already have many new rows with the sharding key set (example).
Proposal
- Implement new base migration job class that move the filter to the sub-batch (example).
- Update Housekeeper to use this new class.
- In addition, add support for custom name for the back-fill table's primary key - !170680 (comment 2184730252).
Updating the existing class could be also an option, but we should make sure this will not have negative effect on existing migrations in progress, as it will change how batch boundaries are calculated.