Geo: Implement snippet repository replication for organization migration
Problem
As part of organization migration from Legacy Cell to Protocell, the Geo team needs to implement special handling for snippet repository replication. The snippet_repositories table requires sharding implementation, schema changes, and custom migration logic to handle organization migration properly.
Background
snippet_repositories are being sharded by organization_id (currently marked severity::4).
Proposal
-
Replace shard_idFK with LFK (loose foreign key) -
Classify as gitlab_main_org(it is currentlygitlab_main_cell_local) -
Define sharding key as group_id. It is already non-nullable.
On migration:
-
Copy snippet_repositoriesrows to the Protocell according to sharding key (as usual). This will be done as part of normal PG replication. -
When copying snippet_repositories, exclude/nullifyshard_idvalues (since they are invalid). If using AWS DMS for PG replication, use a transformation. -
Geo can sync snippet Git repos as usual, with one addition: If shard_idisNULL, then set it with::Repository.pick_storage_shard -
Ensure there are sufficient automated (unit? QA?) tests to meet GitLab's standards -
Smoke test locally
Related
- Parent epic: Organization migration: Handle replicables of c... (&18601)
- Architecture decisions: #515383 (comment 2743597917)+
- Blocked by: gitlab-com/create-stage/source-code-be#287 (snippet sharding)
Edited by Michael Kozono