fix(gitalyctl): moving forked repository next to root

What

When we forked_from_project check that the repository_storage of the root repository is accepting new projects, if it isn't the storage is out of disk space and we should keep the pools fragmented.

Why

In production#17004 (closed) we saw gitalyctl move a lot of forks to storage that didn't have any weight (meaning they are full of disk space). This is because we check forked_form_project.repository_storage and set the destination storage to that value. However, we haven't considered when the fork network becomes too large that it will completely saturate the disk.

The fork network is already fragmented, so we won't be increasing the disk usage but keep the fragmentation.

If one project has about 5TB of forks, it could be worth moving the forked/fork project before other projects. At the moment we spread the repositories across the new Gitaly nodes and after spreading all the data out we go and dump 5TB onto a single node. If we did the forked/fork projects first then the normal projects would assumably be automatically spread out to accommodate for this. However this would make the migration code more complex, and we might want to keep the fragmentation for now especially to spread the workload. cny only has gitaly-org/gitlab and it's hard to operate that Gitaly node since the workload is not balanced and all requests have the same costs which ends up saturation the storage compute.

Testing

On a fresh GDK installation, with multiple git gitaly storages

1. Start rails console

   ./gitlab/bin/rails c

2. Get the first fork network and the projects

   ForkNetwork.first
   => #<ForkNetwork:0x000000013c02cc08 id: 1, root_project_id: 6, deleted_root_project_name: nil>
   
   Project.find(6).repository_storage
   => "default"
   
   Project.find(6).forks.first.repository_storage
   => "default"

3. Configure weights http://gdk.test:3000/admin/application_settings/repository#js-repository-storage-settings to have gitaly-1 to 100, other storages are 0

4. Manually schedule the move for root repository to be in gitaly-1

   curl --request POST --header "PRIVATE-TOKEN: $(op read op://private/GDK/PAT)" "http://gdk.test:3000/api/v4/projects/6/repository_storage_moves"

5. Confirm in the rails console move was successful

   Project.find(6).repository_storage
   => "gitaly-1"

6. Update weights http://gdk.test:3000/admin/application_settings/repository#js-repository-storage-settings to have gitaly-2 to 100, other storages are 0

7. Drain default

   config.yml

   gitlab:
     api_url: http://gdk.test:3000
   
   storage:
     concurrency: 10
     move:
       - default
   
   group:
     concurrency: 100
     move_timeout: 1h
     move_status_update: 5s
   
   snippet:
     concurrency: 100
     move_timeout: 1h
     move_status_update: 5s
   
   project:
     concurrency: 100
     move_timeout: 1h
     move_status_update: 5s

   GITALYCTL_API_TOKEN=$(op read op://private/GDK/PAT) make gitalyctl

8. Validate that forked didn't go to gitaly-1 but went to gitaly-2, and the root is in a separate storage.

   Project.find(6).repository_storage
   => "gitaly-1"
   
   Project.find(6).forks.first.repository_storage
   => "gitaly-2"

Reference: production#17004 (closed)