Store repository host node assignments
Currently Praefect replicates a repository to every storage node within a virtual storage. This can be wasteful and can prevent scaling up large installations as the replication factor increases with the number of Gitaly nodes.
To lay the foundations for the ongoing work of decoupling the replication
factor from the number of Gitaly nodes, this commit adds an assigned
column to the storage_repositories
table. The columns's semantics are
to be as follows:
-
assigned
indicates whether the given storage is an assigned host node of the repository. - On repository creation, records with
assigned
set will be inserted for a number of storages defined by the configured default replication factor. - Replication factor can be increased by upserting a record of a non-assigned storage to be assigned.
- Replication factor can be decreased by removing the assigned flag from
storages until the records with
assigned
set matches the desired replication factor. - Replication will only target nodes that are assigned and requests should only be routed to assigned storages. Replication can originate from unassigned nodes in order to support changing the host nodes and node draining.
- Once the assigned nodes all have a later or equal generation as any unassigned node for a given repository, the unassigned replicas will be targeted for deletion.
- Every repository has to have at least one storage assigned.
Storages which are removed from the cluster but still have records in the database are to be ignored. The configured replication factor is not stored in the initial implementation. That is, if an assigned node of a repository is removed from the cluster, the repository's replication factor decreases. This can be addressed later by storing the desired replication factors as well. Adding new storages to a cluster does not increase a repository's replication factor.
For backwards compatibility during the implementation, assigned
defaults
to true in order retain the existing behavior in non assignment aware code.
If a default replication factor has not been set, it will default to the
number of Gitaly nodes in the virtual storage.
Related to #2971 (closed)