Investigate Geo Failover Zero Downtime Container Registry Migration
Experiment summary
We believe that it's possible to migrate the secondary registry of a geo site to the new container registry metadata database, failover, upgrade the primary, resync, and fallback for a Zero Downtime Container Registry Migration for even the largest registry deployments.
To verify that, we will perform a test doing the procedure described above.
And we’ll measure the impact on total registry downtime during the migration.
Hypothesis
It should be possible to use Geo replication to sync from a non-database backed registry to one using the database. Therefore, the following procedure should be possible for an existing geo registry setup:
- perform the container registry migration process through step one on the secondary site
- break replication
- perform step two
- reestablish replication
- resync
- break replication
- failover to secondary site
- perform the container registry migration process through step two on the secondary site
- reestablish replication
- resync
- failover to primary site
- perform step three on the primary and secondary sites
Similarly, this procedure could be modified for a fresh secondary site with an empty container registry already configured with the metadata database:
- establish replication
- perform the initial sync from the primary
- break replication
- failover to secondary site
- perform the container registry migration process through step two on the secondary site
- reestablish replication
- resync
- failover to primary site
- perform step three on the primary site
Business problem
This process could essentially eliminate the read-only time required for migrating the container registry to the metadata database. While this period is already quite short for three-step migrations, very large registries with high uptime requirements could effectively remove the read-only component in exchange for a more complex procedure.
Supporting data
In extreme cases, we could reduce read-only time from several hours, to a few minutes.
ICE score
| Impact | Confidence | Ease | Score |
|---|---|---|---|
| 3 | 8 | 7 | 6 |
Known assumptions
We assume that Geo replication from an object storage backed container registry to one using the database is possible, stable, and bug free.
Results, lessons learned, next steps
Checklist
- Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start.
- Add the label of the
group::that will work on this experiment (if known). - Mention the Product Manager, Engineering Manager, and at least one Product Designer from the group that owns the part of the product that the experiment will affect.
- Fill in the values in the ICE score table ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the ~"ICE Score Needed" label to indicate that the score is incomplete.
- Replace the ~"ICE Score Needed" with an ICE low/medium/high score label once all values in the ICE table have been added.
- Mention the [at]gitlab-core-team team and ask for their feedback.