Develop a Repositry Migration Coordination Service
Context
As part of #191 (closed) we determined that we needed a service to manage repository imports. This issue is to track the design and development of this service.
General Description
This service will select single repositories from pre-supplied list and pass them off to workers using the import tool directly or lightweight services or goroutines based off the internals of that tool.
Requirements
Core Requirements
- Read a list of repositories from some kind of stable storage since the list is expensive to generate
- Track the state of the provided repositories, so they are not needlessly imported twice, and that proxy registry and redirect requests scoped to this registry appropriately
- Allow failed repository imports to be retried
Good to Have
- Meet the observability requirements outlined here: #191 (closed)
- This service should be able to prioritize or exclude certain repositories for import, possible selection criteria:
- Group
- Number of tags
- Repositories which have had a successful tag cleanup
- Last access time
- Tier
- Dispatch imports to multiple import workers
- Cancel in-progress imports
- Switch repositories to read-only during import
- Coordinate pre-import and import phases
Testing Requirements
Integration Testing
We'll need to ensure we test against a data set large and diverse enough that we can accurately determine that the service is correctly prioritizing repositories. As we add more criteria, we'll need to expand the test data set accordingly.
We'll also need to ensure that failed imports are retried later, but we'll need to ensure that a constantly failing repository does not block the queue.
Testing in Realistic Environments
This service should be included in forthcoming migration tests realistic dataset/workload, and we should make sure this service is adequately covered in those exercises.
Challenges
This service needs to coordinate between the import worker and the proxy registry, so there needs to be a common and reliable means of communication. Especially if we decide to do optimistic locking for writes during repository, in which imports are canceled and rolled back if an incoming write comes in during the import. Most likely this will be the metadata database, but that is currently undetermined.
Even if we do traditional locking during imports, we'll need to make sure the proxy registry does not allow write requests for the duration of the imports.
Additionally, we'll need to make sure that once a repository has been successfully imported, the proxy registry sends all traffic for this repository to the new registry. If there are writes to the original filesystem metadata of the repository, it's very likely we'll have divergent datasets that cannot be automatically reconciled, so we should take special care to avoid this.