Rebalance some storage shards
Production Change - Criticality 2 C2
| Rebalance some storage shards |
Prevent our storage servers from running out of space |
| Change Type |
C2 |
| Services Impacted |
GitLab.com |
| Change Team Members |
@skarbek |
| Change Severity |
How critical is the change |
| Buddy check |
A colleague will review the change |
| Tested in staging |
Kinda... |
| Schedule of the change |
TBD |
| Duration of the change |
Time to execute the change ( including a possible rollback ) |
| Detailed steps for the change. Each step must include: |
See below |
Procedure
- scp the storage script gitlab-com/runbooks!878 (merged) into a path where the git user has access
scp storage_rebalance.rb console-01-sv-gprd.c.gitlab-production.internal:/tmp/
- Start by logging into the console server
- This is a repetitive procedure that will repeat PER server using the below table as substitutes
Steps
sudo gitlab-rails runner /tmp/storage_rebalance.rb --current-file-server <SERVERNAME> --target-file-server <SERVERNAME> --move-amount <MOVE_AMOUNT> --dry-run false --wait 600 | tee <current-file-server-name>.log
Movement Table
| Current File Server |
Target File Server |
Move Amount |
Run |
| nfs-file13 |
nfs-file25 |
1600 |
✅ |
| nfs-file14 |
nfs-file26 |
2080 |
✅ |
| nfs-file15 |
nfs-file27 |
1280 |
✅ |
| nfs-file16 |
nfs-file28 |
2240 |
✅ |
| nfs-file17 |
nfs-file25 |
1760 |
✅ |
| nfs-file18 |
nfs-file26 |
2400 |
✅ |
| nfs-file19 |
nfs-file27 |
1600 |
✅ |
| nfs-file20 |
nfs-file28 |
1760 |
✅ |
| nfs-file21 |
nfs-file25 |
1920 |
❗ |
| nfs-file22 |
nfs-file26 |
3520 |
✅ |
| nfs-file23 |
nfs-file27 |
2560 |
✅ |
| nfs-file24 |
nfs-file28 |
3040 |
✅ |
- nfs-file25 will have approximately 44% used space
- nfs-file26 will have approximately 61% used space
- nfs-file27 will have approximately 45% used space
- nfs-file28 will have approximately 55% used space
/label C2 ~S2 change
Edited
by
John Skarbek