Skip to content

Replicate object pools on repository move

Vasilii Iakliushin requested to merge 412407_replicate_object_pools into master

What does this MR do and why?

Contributes to #412407 (closed)

Problem

When projects are moved from shard to shard, we don't move related object pools with them. That prevents repositories from utilizing object pool features.

Solution

Migrate object pools to new shards if they don't exist there. There are several scenarios possible.

Project doesn't have an object pool

  • Skip object pool migration and move only project repository

Project has an object pool

  1. Shard where project is migrated doesn't have an object pool
  • Copy the object pool to the new shard
  • Create a PoolRepository object in database for the new shard
  • Link the project to the new PoolRepository
  • Link the project to the new object pool
  • Unlink the project from old pool repository
  1. Shard where project is migrated already has an object pool
  • Find PoolRepository object
  • Link the project to the PoolRepository
  • Link the project to the object pool
  • Unlink the project from old pool repository

How to set up and validate locally

  1. Enable feature flag Feature.enable(:replicate_object_pool_on_move)
  2. Configure multiple Gitaly storages. It is possible to do it via GDK: https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/gitaly.md#add-gitaly-nodes
  3. Create a public project, then fork the project in order to create pool_repository
  4. You should see a new record in pool_repositories table (PoolRepository.last should have a source_project link to the original project)
  5. You should see a new folder for the object pool in your GDK (path to pool folder repositories/@cluster/pools/)
  6. Call an API request to move a repository to a different storage: https://docs.gitlab.com/ee/api/project_repository_storage_moves.html#schedule-a-repository-storage-move-for-a-project
  7. You should see a new in pool_repositories table (a new PoolRepository should still point to the same source_project, but it should have a different shard value)
  8. You should see a new folder for the object pool in your GDK (path to pool folder repository_storages/praefect-gitaly-1/praefect-internal-1/@cluster/pools/)
  9. Verify that object pool is replicated as well. Content of related object pools (in repositories/@cluster/pools/ and repository_storages/praefect-gitaly-1/praefect-internal-1/@cluster/pools/) should be the same.
  10. Repeat the migration process for the forked project
  11. Original PoolRepository record should disappear, because all projects were moved to the new shard
  12. Try to migrate project back but in a different order (forked project first, main project next)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasilii Iakliushin

Merge request reports