Skip to content

Store repository host node assignments

Sami Hiltunen requested to merge smh-variable-replication-factor into master

Currently Praefect replicates a repository to every storage node within a virtual storage. This can be wasteful and can prevent scaling up large installations as the replication factor increases with the number of Gitaly nodes.

To lay the foundations for the ongoing work of decoupling the replication factor from the number of Gitaly nodes, this commit adds an assigned column to the storage_repositories table. The columns's semantics are to be as follows:

  1. assigned indicates whether the given storage is an assigned host node of the repository.
  2. On repository creation, records with assigned set will be inserted for a number of storages defined by the configured default replication factor.
  3. Replication factor can be increased by upserting a record of a non-assigned storage to be assigned.
  4. Replication factor can be decreased by removing the assigned flag from storages until the records with assigned set matches the desired replication factor.
  5. Replication will only target nodes that are assigned and requests should only be routed to assigned storages. Replication can originate from unassigned nodes in order to support changing the host nodes and node draining.
  6. Once the assigned nodes all have a later or equal generation as any unassigned node for a given repository, the unassigned replicas will be targeted for deletion.
  7. Every repository has to have at least one storage assigned.

Storages which are removed from the cluster but still have records in the database are to be ignored. The configured replication factor is not stored in the initial implementation. That is, if an assigned node of a repository is removed from the cluster, the repository's replication factor decreases. This can be addressed later by storing the desired replication factors as well. Adding new storages to a cluster does not increase a repository's replication factor.

For backwards compatibility during the implementation, assigned defaults to true in order retain the existing behavior in non assignment aware code. If a default replication factor has not been set, it will default to the number of Gitaly nodes in the virtual storage.

Related to #2971 (closed)

Merge request reports