Enable puma on 10 web and api nodes
C3
Production Change - Criticality 3Change Objective | Describe the objective of the change |
---|---|
Change Type | ConfigurationChange |
Services Impacted | List services |
Change Team Members | Name of the engineers involved in the change |
Change Severity | C3 |
Change Reviewer or tested in staging | A colleague who will review the change or evidence the change was tested on staging environment |
Dry-run output | If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result |
Due Date | Date and time in UTC timezone for the execution of the change, if possible add the local timezone of the engineer executing the change |
Time tracking | To estimate and record times associated with changes ( including a possible rollback ) |
Summary
In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8334 after examining puma on the production canary fleet and web-01/web-02 we have decided to move forward with enabling puma on larger set of web and api hosts. This will be the next step and probably final step in determining whether we should move forward with a full puma rollout across the entire fleet.
This change issue is to enable puma on 10 web and 10 api nodes. We will do this with single node overrides which will be removed when the analysis is complete.
Monitoring
Through the execution of the script we will be monitoring the following dashboards
- web https://dashboards.gitlab.net/d/web-main/web-overview?orgId=1&%3ForgId=1&from=now-1h&to=now
- api https://dashboards.gitlab.net/d/api-main/api-overview?orgId=1&from=now-1h&to=now
Detailed steps for the change
-
Set puma node overrides on host api-01 to api-10 by executing the following script, the script safely removes each VM from the load balancer, issues a node override, and then adds it back.
project="gitlab-production"
env="gprd"
type="api"
for num in $(seq -f "%02g" 10); do
host="$type-$num-sv-$env"
fqdn=$host.c.$project.internal
echo "$host / $fqdn"
echo "---------------------------------------------"
echo ""
./bin/set-server-state -f $env drain $host
knife node attribute set $fqdn . '{ "omnibus-gitlab": { "gitlab_rb": { "puma": { "enable": true }, "unicorn": { "enable": false } } } }'
knife ssh -e "name:$fqdn" "sudo chef-client"
./bin/set-server-state -f $env ready $host
knife ssh "name:$fqdn" "sudo gitlab-ctl status"
done
-
Set puma node overrides on host web-01 to web-10 by executing the following script
project="gitlab-production"
env="gprd"
type="web"
for num in $(seq -f "%02g" 10); do
host="$type-$num-sv-$env"
fqdn=$host.c.$project.internal
echo "$host / $fqdn"
echo "---------------------------------------------"
echo ""
./bin/set-server-state -f $env drain $host
knife node attribute set $fqdn . '{ "omnibus-gitlab": { "gitlab_rb": { "puma": { "enable": true }, "unicorn": { "enable": false } } } }'
knife ssh -e "name:$fqdn" "sudo chef-client"
./bin/set-server-state -f $env ready $host
knife ssh "name:$fqdn" "sudo gitlab-ctl status"
done
rollback
-
Remove node overrides from web-01 - web-10
knife node edit ...
-
Remove node overrides from api-01 to api-10
knife node edit ...
Rollback steps
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
Person on-call has been informed prior to change being rolled out
Edited by John Jarvis