Set PUMA_INJECT_WAIT_TICKS=0 on 2 web and 2 api nodes
C3
Production Change - Criticality 3Change Objective | Describe the objective of the change |
---|---|
Change Type | ConfigurationChange |
Services Impacted | List services |
Change Team Members | Name of the engineers involved in the change |
Change Severity | C3 |
Change Reviewer or tested in staging | A colleague who will review the change or evidence the change was tested on staging environment |
Dry-run output | If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result |
Due Date | Date and time in UTC timezone for the execution of the change, if possible add the local timezone of the engineer executing the change |
Time tracking | To estimate and record times associated with changes ( including a possible rollback ) |
Overview
The last step for enabling puma across the entire fleet is to verify the difference between the patched and unpatched puma on production. We previously executed this change and collected data with #1619 (closed) . This is a repeat of that for a 2 hour interval.
This validation was discussed as part of the puma performance tuning https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8334#note_279949673 with @ayufan and @andrewn
Detailed steps for the change
This has already been tested on staging using the same script below
-
10:30UTC Run the following command to apply the env update and run chef-client
#!/bin/bash
set -e
env=gprd
project=gitlab-production
for host in web-01-sv-$env web-02-sv-$env api-01-sv-$env api-02-sv-$env; do
fqdn="$host.c.$project.internal"
echo "$fqdn"
echo "-------"
echo ""
knife node attribute set $fqdn . '{ "omnibus-gitlab": { "gitlab_rb": { "gitlab-rails": { "env": { "PUMA_INJECT_WAIT_TICKS": "0" } } } } }'
knife ssh -e "name:$fqdn" "sudo chef-client"
done
-
Keep the overrides in place and validate and compare latency between nodes that have the variable set and thos that do not. -
12:00UTC Remove the overrides
Rollback steps
- Remove the overrides
for host in web-01-sv-$env web-02-sv-$env api-01-sv-$env api-02-sv-$env; do
echo "$fqdn"
echo "-------"
echo ""
fqdn="$host.c.$project.internal"
knife node edit $fqdn
knife ssh -e "name:$fqdn" "sudo chef-client"
end
Changes checklist
-
Detailed steps and rollback steps have been filled prior to commencing work -
Person on-call has been informed prior to change being rolled out
Edited by John Jarvis