Validate SmartHTTP::PostUploadPack
~Conversation: #219 (closed)
See the Migration Process documentation for more information on the Acceptance Testing stage of the process.
Feature Toggle Environment Variable: GITALY_POST_UPLOAD_PACK
-
Chef recipes to enable/disable this feature https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/684 -
Grafana dashboard for monitoring gitaly-dashboards!5 (merged) -
Environments -
dev.gitlab.org
https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/685 -
Staging https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/684 -
gitlab.com
https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/696 https://docs.google.com/document/d/1SmZvlT7D1dXsEtRPZc0xWR0jGmPmLztlGkG4_L9DQeU/edit
-
-
Test Results (post as comments on this issue) -
Did the migration perform as expected? -
Did the code have reasonable performance characteristics? -
Did error rates jump to an unacceptable level?
-
-
Have the changes been rolled back pending final review? -
Runbook Added (link to MR) -
Prometheus Alerts Added (link to MR)
We now (9.1) have a feature flag GITALY_POST_UPLOAD_PACK in gitlab-ce. We should validate that it works:
-
test in omnibus nightly -
test in RC omnibus package -
test in staging (apply setting, do manual clones, look at prometheus counters for errors and to see if the RPC is being used) -
test in production for a few hours (same as staging) -
test in production for a week -
enable feature flag by default in omnibus
Staging steps
- update
gitlab-staging-worker
role (make sure change is on chef.gitlab.com, not just in the chef-repo) -
knife ssh -C 1 role:gitlab-staging-worker 'sudo chef-client; sleep 60; sudo gitlab-ctl term unicorn; echo done $?'
(-C 1
is probably too slow for production) knife ssh role:gitlab-staging-worker 'for p in $(pgrep -f "unicorn master"); do ps -o pid,etime,args -p $p; sudo cat /proc/$p/environ | xargs --null --max-args=1; done' | grep GITALY
- check gitaly staging prometheus dashbord
- check Sentry for staging
- manual tests
- revert config change
Local omnibus steps
In gitlab.rb:
gitlab_rails['env'] = {'GITALY_POST_UPLOAD_PACK' => '1'}
To apply or revert:
sudo gitlab-ctl reconfigure
sleep 60 # wait for unicorn to reload
sudo gitlab-ctl term unicorn
# test environment of unicorn master. should print a match
for p in $(pgrep -f 'unicorn master'); do ps -o pid,etime,args -p $p; sudo cat /proc/$p/environ | xargs --null --max-args=1 | grep GITALY; done
The rigmarole with gitlab-ctl term
is because of omnibus-gitlab#2002
Edited by Jacob Vosmaer