Increase unicorn workers on git frontend nodes

Production Change - Criticality 3 C3

Change Objective	Increase the unicorn workers on git frontend nodes to see if it helps reduce queuing for ssh auth requests, without causing negative impact
Change Type	ConfigurationChange
Services Impacted	git-over-http, and git-over-ssh authentication
Change Team Members	@cmiskell
Change Severity	C3 - numerical worker count change only, easily reversible
Buddy check or tested in staging	Requires review (TBD); change on staging will not show any useful effects due to negligible load there
Schedule of the change	2019-09-11 02:45 UTC
Duration of the change	30 minutes (60 if we have to rollback)
Detailed steps for the change. Each step must include:	Below - pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step

1. Stop chef on the git servers: `knife ssh 'roles:gprd-base-fe-git' "sudo systemctl stop chef-client"
2. Merge and apply the MR (https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1692) that will bump the unicorn worker count from 25 to 30 on the front-end git servers
3. Run chef-client on one git node (git-cny-01-sv-gprd.c.gitlab-production.internal) and then verify that we now have 30 workers with ps auwx|grep "unicorn worker"|grep -v grep | wc -l. If not, determine what other steps may be required (sudo gitlab-ctl hup unicorn, perhaps)
4. Apply the change by converging chef on all nodes in this role, with some parallelism but constrained to avoid restarting all the unicorns at once: knife ssh -C4 'roles:gprd-base-fe-git' "sudo chef-client"
5. Monitor
   1. https://prometheus-app.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=unicorn_queued_connections%7Bsocket_type%3D%22tcp%22%2Csocket_address%3D%220.0.0.0%3A8080%22%2Ctype%3D%22git%22%7D&g0.tab=0 (and the sum of that) around an hour boundary. We are expecting reduced numbers of queued connections, although we're not exactly sure how pronounced the effect will be. Our goal is zero, but we want to approach that carefully.
   2. https://dashboards.gitlab.net/d/mnbqU9Smz/fleet-overview, specifically the git workers cpu and load graphs. We do not expect either to rise much, but they may a little. Large rises (10-20% real CPU) would be a marker to roll back.
   3. https://dashboards.gitlab.net/d/Qe6veT_mk/fleet-utilization?orgId=1&refresh=5m&fullscreen&panelId=12 . Memory usage will grow, but given the current usage is in the 20s (%) we've got a lot of wiggle room and this would have to completely unexpectedly explode (e.g. to over 70%) for there to be any reason to roll back.
   4. Let it sit for 1 day, and increase by 5 every day until the queuing disappears or other symptoms emerge that suggest going back. The initial goal is 40 workers, with an instinct that 60 is probably too much.

Rollback

1. Revert the commit in https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1692
2. Apply the change by converging chef on all nodes in this role, with some parallelism but constrained to avoid restarting all the unicorns at once: `knife ssh -C4 'roles:gprd-base-fe-git' "sudo chef-client"

Context: Thread at https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7168#note_193491217