In the past, Praefect did not create a database records for hosted
repositories. Nowadays Praefect tracks repositories and the storage's
hosting copies of them in the repositories and the storage_repositorie
tables. Various components in Praefect, including the automatic reconciler,
operate only based on these database records. As such, repositories which
have received no mutators after the introduction of the tables are ignored.
There are two work in progress features in the pipeline that also depend on the database records.
- Repository specific primaries need to know which repositories exist in order to elect a primary for them.
- Variable replication factor needs to know which storage locations contain the repository in order to route the requests to correct nodes.
This commit adds support for creating database records for repositories which are still missing them. The importer walks the primary storage of each virtual storage and creates a record for each repository existing on the primary. The created records can then be picked up by the automatic reconciler which will replicate the repositories to the secondaries and create their records.
Repositories on secondaries are ignored by the importer. This matches how Praefect would behave in general. If a repository is missing database records and it receives a write, Praefect follows it's previous behavior and simply accepts the write and overwrites the copies on the secondaries.
Once the migration has been successfully completed, a flag in the database is set to prevent the migration from running again unnecessarily. If the importer does not run to completion, it is safe to rerun. Running the importer concurrently is also safe as only missing records are added and existing ones are unmodified.
The import runs on each Praefect node on start up and retries on every restart until it has successfully completed.
Closes #3033 (closed)