S Add sharding key YAML config to ci_runner_machines

What we need to do

The ci_runner_machines has runner_id, so should ensure that ci_runner_machines uses the same sharding key as ci_runners.

How to do it

1. #497526 (closed): Add partitioning column runner_type.
   1. Backfill runner_type
      • Update the app to populate runner_type for new runner managers
   2. Post-backfill cleanup (Release N+1)
2. #497526 (closed): Add sharding_key_id column to ci_runners table
   1. Backfill sharding_key_id
      • Update the app to populate sharding_key_id for new runner managers
   2. !169964 (merged): Post-backfill cleanup (Release N+1)
   3. Optional if we take the approach of only copying valid records to the partitioned table.
      1. Delete invalid records.
      2. Validate sharding key constraints.
3. #497562 (closed): Create routing table and its 3 partitions:
   1. Create p_ci_runner_managers routing table, and its 3 partitions:
      • instance_type_ci_runner_managers will be attached to partition 1, and will be exempt from sharding.
      • group_type_ci_runner_managers will be attached to partition 2, and will be sharded by namespace: namespace_id.
      • project_type_ci_runner_managers will be attached to partition 3, and will be sharded by project: project_id.
   2. Mimic LFK definitions from ci_runner_machines in config/gitlab_loose_foreign_keys.yml for the new p_ci_runner_machines.
   3. Create triggers to keep ci_runner_machines and each of the partitioned tables in sync.
4. Copy data to partitioned table (enqueue_partitioning_data_migration).
   1. Backfill ci_runner_machines_687967fa8a table (#502934 - closed). This is estimated to take less than half an hour on .com.
   • after next required stop (%17.9):
     1. Finalize BackfillCiRunnerMachinesPartitionedTab... (#504291 - closed)
5. Delete records on ci_runner_machines_687967fa8a that don't exist in ci_runners_e59bb2812d, so we can validate the fk_rails_3f92913d27 FK constraint.
   • .com (%17.7):
     1. Delete orphaned records on ci_runner_machines_6... (#503748 - closed)
   • non-.com (%17.9):
     1. Delete orphaned records on ci_runner_machines_6... (#504297 - closed)
6. Recreate foreign key constraints from ci_runner... (#502403 - closed) (%17.8)
7. #504965 (closed): Call replace_with_partitioned_table and set Ci::RunnerManager.primary_key = [:runner_type, :id].
8. Mark db/docs/ci_runner_machines.yml as exempt_from_sharding: true. This will affect Org mover as it is planned to skip exempt_from_sharding tables from org mover.
9. Delete DeleteOrphanedPartitionedCiRunnerMachine... (#504282 - closed), non-.com (%18.0)
10. #516060 (closed): Once everything is verified to be running as expected, drop ci_runner_machines table.
11. #503749 (closed): Validate fk_rails_3f92913d27 FK constraint (%18.3).
12. Add organization_id column to runner tables (#523694 - closed)
13. Add logic to validate organization_id field in ... (#548139 - closed)
14. Finalize organization_id backfill migrations (#523850 - closed)
15. Validate fk_rails_3f92913d27 FK constraint on c... (#503749 - closed)
16. Finalize reintroduced BBM that deleted orphaned... (#530612 - closed)
17. Add check_organization_id_nullness constraint c... (#523851 - closed)
18. Validate check_organization_id_nullness constra... (#523852 - closed)
19. Column organization_id on runner tables should ... (#525293 - closed)
20. Ignore sharding_key_id column in runner models (#547654 - closed)
21. Drop sharding_key_id column from runner tables (#547650 - closed) ...