Zero-Downtime Phased Migration Proposal for Large Repositories
Introduction
This issue discusses a strategy in which large repositories may be migrated with zero downtime. This technique will require some level of development on the registry itself to support this approach.
🤔 Assumptions
We do not wish to run the current iteration of garbage collection with downtime against large registries using Manifest Lists due to #149 (closed). Meaning that to clean up registry storage while preserving the integrity of manifest lists, we must first import all metadata into the database.
Large deployments running several instances of the registry may wish to gradually rollout changes, for example only having 10% of requests going to a database backed registry initially.
New writes are rare: roughly 2.3% of requests
💡 Migration Strategy
🗄 Phase 0 — Filesystem Only
This is the registry as it is deployed today — all metadata accessed through a filesystem or object storage.
♻ Phase 1 — Write Mirroring with Filesystem Read Failback
Incoming writes are mirrored to both the filesystem as well as the database. Write mirroring currently works with the completion of &3006 (closed)
Reads are first tried from the database, in the case of a not found error we will look for the data in the filesystem.
Once all instances of the registry are configured for this, all new writes are guaranteed to be present in the database.
🔍 Phase 2 — Metadata Backfilling
In this phase the Import Command is ran against the registry storage. Once this is complete, all metadata should be contained in the database.
🐘 Phase 3 — Write Mirroring with Database Only Metadata Reads
In this phase, the registry will be configured to read metadata exclusively from the database. Writes make up such a small proportion of requests that continuing to mirror them is worth the additional costs to enable a gradual rollout.
This phase is needed for two reasons:
- A registry in a Phase 1 configuration tries failback to the filesystem even if the entity is truly not known to the registry, this introduces a performance impact that should be avoided.
- Without confidence that 100% of the metadata is represented in the filesytem, we can only try to read from the database when we're addressing a single entity. Actions like listing all tags must use the filesystem in order to list the full results.
📈 Advantages
- Each phase is relatively forgiving.
- Phase 1 could be gradually rolled out along side of registries in Phase 0 reducing the risk for large deployments.
- The process can be completely rolled back through the end of Phase 3.
- Like phase 1, phase 3 is also able to be gradually rolled out.
- We are able to incrementally gain benefits from the database starting from phase 1. Reads for newly written data will be able to use the database immediately and as phase 2 progresses a larger portion of reads for older data will be able to be requested via the database.
- No downtime at all is required
- Continuing to mirror writes to the filesystem prevents users from being locked into using our registry implementation.
- Preserving the original filesystem metadata storage allows smaller deployments to avoid the complexity of the database if they do not need it.
📉 Disadvantages
- A non-insignificant amount of registry development will need to be undertaken to enable the non-trivial behavior described in phase 1.
- The importer was not designed with this usage in mind, and would need to be some development to work with an existing database as well as implementing a flag to import metadata about dangling blobs
- A full run of the importer for Phase 2 could potentially last several weeks for the very largest registries, a form of check pointing may need to be implemented in order not to resume from the beginning
- Phase 1 must be 100% complete for phase to begin and similar for phase 2 and 3.
- Once phase 1 is complete, phase 0 registries can not connected to the filesystem storage (at least with writes enabled) or the process must resume from the start to guarantee a fully consistent metadata database. This would not affect the stability of the deployment, however.
- We will not be able to remove the filesystem metadata code.
🗣 Proposal
-
Discuss feasibility of this proposal with the package team -
Discuss feasibility of this proposal with the infrastructure team -
Implement the phase 1 and 3 behaviors using the existing services, distribution.ManifestStore
, etc. This should allow phase 0, 1, and 3 code to exist side-by-side and be easy to configure -
Implement #87 (closed) -
Adjust the importer to be able to work with a non-fresh database -
Develop a checkpointing feature for the import tool