Migrate large projects off file-28-stor-gprd to file-38-stor-gprd
What?
Migrate the large projects currently on file-28-stor-gprd.c.gitlab-production.internal
(https://dashboards.gitlab.net/d/W_Pbu9Smk/storage-stats?orgId=1&refresh=30m&fullscreen&panelId=138) to file-38-stor-gprd.c.gitlab-production.internal
(https://dashboards.gitlab.net/d/W_Pbu9Smk/storage-stats?orgId=1&refresh=30m&fullscreen&panelId=148)
Why?
Because git data file server disk usage is high: 81.90% full.
We need to build new file servers or rebalance repos.
Since we have four other file storage nodes with more than 50% capacity available, a rebalance instead of additional node creation, seems optimal.
Procedure
-
Install the rebalance script on a common system (such as the production console system host).
ssh console-01-sv-gprd.c.gitlab-production.internal
tmux attach || tmux # attach to an existing tmux session or create a new one
sudo su - root
mkdir -p /var/opt/gitlab/scripts
chown -R git:git /var/opt/gitlab/scripts
cd /var/opt/gitlab/scripts
curl --location --silent --remote-name https://gitlab.com/gitlab-com/runbooks/raw/master/scripts/storage_rebalance.rb
chown -R git:git /var/opt/gitlab/scripts/storage_rebalance.rb
chmod +x storage_rebalance.rb
gitlab-rails runner /var/opt/gitlab/scripts/storage_rebalance.rb --help
-
Export a personal access token in an environment variable on the console system host.
export PRIVATE_TOKEN=CHANGEME
Dry-run
-
Execute a dry-run of the rebalance script to move files from nfs-file28
tonfs-file38
. Do not give an amount of disk space to migrate (--move-amount N
), so that only one single project will be migrated for the time being.
gitlab-rails runner /var/opt/gitlab/scripts/storage_rebalance.rb --verbose --current-file-server=nfs-file28 --target-file-server=nfs-file38 --wait=10800 --dry-run=yes
-
Verify that the dry-run execution behaved as expected (no errors).
Don't jump in the deep end
-
Take a snapshot of the disk for nfs-file28
. Navigate to https://console.cloud.google.com/compute/disksDetail/zones/us-east1-c/disks/file-28-stor-gprd-data?project=gitlab-production in a browser. Click the "Create Snapshot" button, and then click the "Create" button. Or the following commands may be used in a shell session on your local workstation:
gcloud auth login
gcloud config set project gitlab-production
gcloud config set compute/region us-east1
gcloud config set compute/zone us-east1-c
gcloud compute disks list | grep file-28-stor-gprd-data
gcloud compute disks snapshot file-28-stor-gprd-data
-
Execute the rebalance script to move files from nfs-file28
tonfs-file38
. Do not give an amount of disk space to migrate (--move-amount N
), so that only one single project will be migrated for the time being. Specify that the rebalance script will wait much longer than 10 seconds, to ensure that the script continues to monitor the progress of the project migration until it has completed.
gitlab-rails runner /var/opt/gitlab/scripts/storage_rebalance.rb --dry-run=no --verbose --current-file-server=nfs-file28 --target-file-server=nfs-file38 --wait=10800 | tee nfs-file28.migration.$(date +%Y-%m-%d_%H%M).log
-
Verify that the execution behaved as expected (no errors).
Wade in
-
Execute the rebalance script to move files from nfs-file28
tonfs-file38
. Specify an amount of disk space in gigabytes (less than 16000) to migrate.
gitlab-rails runner /var/opt/gitlab/scripts/storage_rebalance.rb --dry-run=no --verbose --current-file-server=nfs-file28 --target-file-server=nfs-file38 --wait=10800 --move-amount=1000 | tee nfs-file28.migration.$(date +%Y-%m-%d_%H%M).log
-
Verify that the execution behaved as expected (no errors).
Edited by Nels Nelson