fix(gitalyctl): moving forked repository next to root
What
When we forked_from_project
check that the repository_storage of the
root repository is accepting new projects, if it isn't the storage is
out of disk space and we should keep the pools fragmented.
Why
In production#17004 (closed) we
saw gitalyctl
move a lot of forks to storage that didn't have any
weight (meaning they are full of disk space). This is because we check
forked_form_project.repository_storage
and set the destination storage
to that value. However, we haven't considered when the fork network
becomes too large that it will completely saturate the disk.
The fork network is already fragmented, so we won't be increasing the disk usage but keep the fragmentation.
If one project has about 5TB of forks, it could be worth moving
the forked/fork project before other projects. At the moment we spread the
repositories across the new Gitaly nodes and after spreading all the
data out we go and dump 5TB onto a single node. If we did the
forked/fork projects first then the normal projects would assumably be
automatically spread out to accommodate for this. However this would
make the migration code more complex, and we might want to keep the
fragmentation for now especially to spread the workload. cny
only has
gitaly-org/gitlab
and it's hard to operate that Gitaly node since the
workload is not balanced and all requests have the same costs which ends
up saturation the storage compute.
Testing
On a fresh GDK installation, with multiple git gitaly storages
-
Start rails console
./gitlab/bin/rails c
-
Get the first fork network and the projects
ForkNetwork.first => #<ForkNetwork:0x000000013c02cc08 id: 1, root_project_id: 6, deleted_root_project_name: nil> Project.find(6).repository_storage => "default" Project.find(6).forks.first.repository_storage => "default"
-
Configure weights http://gdk.test:3000/admin/application_settings/repository#js-repository-storage-settings to have
gitaly-1
to 100, other storages are0
-
Manually schedule the move for root repository to be in
gitaly-1
curl --request POST --header "PRIVATE-TOKEN: $(op read op://private/GDK/PAT)" "http://gdk.test:3000/api/v4/projects/6/repository_storage_moves"
-
Confirm in the rails console move was successful
Project.find(6).repository_storage => "gitaly-1"
-
Update weights http://gdk.test:3000/admin/application_settings/repository#js-repository-storage-settings to have
gitaly-2
to100
, other storages are0
-
Drain
default
config.yml
gitlab: api_url: http://gdk.test:3000 storage: concurrency: 10 move: - default group: concurrency: 100 move_timeout: 1h move_status_update: 5s snippet: concurrency: 100 move_timeout: 1h move_status_update: 5s project: concurrency: 100 move_timeout: 1h move_status_update: 5s
GITALYCTL_API_TOKEN=$(op read op://private/GDK/PAT) make gitalyctl
-
Validate that forked didn't go to
gitaly-1
but went togitaly-2
, and the root is in a separate storage.Project.find(6).repository_storage => "gitaly-1" Project.find(6).forks.first.repository_storage => "gitaly-2"
Reference: production#17004 (closed)