Skip to content

Add and backfill a sharding_key to an existing large table

Similar to #429173 (closed) but we want to work out the details needed for migrating a very large table.

A candidate table might be security_findings but this table is huge so it may even be too big.

Possible tables

  1. security_findings
    1. Started in !135691 (closed) but blocked by #431134 (closed)
  2. label_links
  3. resource_label_events
  4. merge_request_diffs

merge_request_diffs.project_id:

  1. MR1 !136800 (merged):
    1. Adds the column in db/migrate
    2. Adds the application code to write the column
    3. Adds the invalid not-null constraint in db/post_migrate
    4. Adds the async index on project_id
  2. MR2: !139726 (merged): (once all old records without project_id have been updated)
    1. Finalize the batched background migration
    2. Validates the not-null constraint
  3. MR3 !137495 (merged): (when the async index creation is finished)
    1. Finalizes the async index creation
    2. Adds the foreign key invalid
  4. MR4 !140410 (merged):
    1. Adds the async validation of the foreign key constraint
  5. MR5 !140280 (merged):
    1. Finalizes validating the foreign key constraint
Edited by Tiger Watson