Validate SmartHTTP::PostUploadPack

~Conversation: #219 (closed)

See the Migration Process documentation for more information on the Acceptance Testing stage of the process.

Feature Toggle Environment Variable: GITALY_POST_UPLOAD_PACK

---

• Chef recipes to enable/disable this feature https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/684
• Grafana dashboard for monitoring gitaly-dashboards!5 (merged)
• Environments
  • dev.gitlab.org https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/685
  • Staging https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/684
  • gitlab.com https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/696 https://docs.google.com/document/d/1SmZvlT7D1dXsEtRPZc0xWR0jGmPmLztlGkG4_L9DQeU/edit
• Test Results (post as comments on this issue)
  • Did the migration perform as expected?
  • Did the code have reasonable performance characteristics?
  • Did error rates jump to an unacceptable level?
• Have the changes been rolled back pending final review?
• Runbook Added (link to MR)
• Prometheus Alerts Added (link to MR)

---

We now (9.1) have a feature flag GITALY_POST_UPLOAD_PACK in gitlab-ce. We should validate that it works:

• test in omnibus nightly
• test in RC omnibus package
• test in staging (apply setting, do manual clones, look at prometheus counters for errors and to see if the RPC is being used)
• test in production for a few hours (same as staging)
• test in production for a week
• enable feature flag by default in omnibus

---

Staging steps

• update gitlab-staging-worker role (make sure change is on chef.gitlab.com, not just in the chef-repo)
• knife ssh -C 1 role:gitlab-staging-worker 'sudo chef-client; sleep 60; sudo gitlab-ctl term unicorn; echo done $?' (-C 1 is probably too slow for production)
• knife ssh role:gitlab-staging-worker 'for p in $(pgrep -f "unicorn master"); do ps -o pid,etime,args -p $p; sudo cat /proc/$p/environ | xargs --null --max-args=1; done' | grep GITALY
• check gitaly staging prometheus dashbord
• check Sentry for staging
• manual tests
• revert config change

Local omnibus steps

In gitlab.rb:

gitlab_rails['env'] = {'GITALY_POST_UPLOAD_PACK' => '1'}

To apply or revert:

sudo gitlab-ctl reconfigure
sleep 60 # wait for unicorn to reload
sudo gitlab-ctl term unicorn
# test environment of unicorn master. should print a match
for p in $(pgrep -f 'unicorn master'); do ps -o pid,etime,args -p $p; sudo cat  /proc/$p/environ | xargs --null --max-args=1 | grep GITALY; done

The rigmarole with gitlab-ctl term is because of omnibus-gitlab#2002