Expand canary to include more paths

Production Change - Criticality 3 C3

Change Objective	Describe the objective of the change
Change Type	Expand canary to include more paths
Services Impacted	Canary
Change Team Members	@jarv
Change Severity	C3
Buddy check or tested in staging	@mwasilewski-gitlab
Schedule of the change	Mon Oct 7 11:00:00 UTC 2019
Duration of the change	20min
Detailed steps for the change. Each step must include:	- pre-conditions for execution of the step, - execution commands for the step, - post-execution validation for the step , - rollback of the step

We currently route the following request paths to canary:

"path_list": [
            "/charts",
            "/groups/gitlab-com",
            "/gitlab-com",
            "/gitlab-org/gitlab-ee",
            "/api/v4/groups/gitlab-com",
            "/api/v4/projects/gitlab-com",
            "/api/v4/projects/gitlab-org",
            "/api/v4/groups/gitlab-org"
          ]

For testing puma, we would like to add a bit more traffic to the canary stage, specifically we will be adding/modifying the following paths:

/groups/gitlab-org
/gitlab-org
/gitlab-data
/groups/gitlab-data
/meltano
/groups/meltano

• at peak, the canary stage currently receives ~120 req/second, compared to the main stage which receives ~3000 req/second
• Per node, canary receives about half the traffic on a typical web node
• cpu for the canary stage peaks at 25%, for the main stage we peak at ~50%
• memory peaks at 20GB which is well below our available memory which is 60GB

Part 1

• Merge https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1940 to add more GitLab groups to the canary path
• Monitor the canary dashboard and per-node request rates

Rollback

• Disable canary traffic /chatops run canary --drain --production
• Revert https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1940

Part 2 TBD

• if there is still not enough traffic we will be adding projects suggested in https://gitlab.com/gitlab-com/gl-infra/production/snippets/1901754

Monitoring

• Dashboard to monitor per node request rates - https://dashboards.gitlab.net/d/mO9noY2Wz/increase-canary-traffic?orgId=1&from=now-6h&to=now
• Canary dashboard - https://dashboards.gitlab.net/d/llfd4b2ik/canary?orgId=1
• General service metrics - https://dashboards.gitlab.net/d/general-service-stages/general-service-platform-metrics-stages?orgId=1

added C3 ~7170326 labels

assigned to @jarv

As discussed in the puma timeline sync, the goal should be to bring up the req/sec on the canary nodes up to the same number of req/sec on main stage, per node https://docs.google.com/document/d/1wXHYLASFa1IP_TtvydCJmytJHEX1FX9GK5yc1vo-rg4/edit

mentioned in epic &78 (closed)

added to epic &78 (closed)

@jarv some candidates as discussed in the call: https://gitlab.com/gitlab-com/gl-infra/production/snippets/1901754

thanks!

One-off dashboard for workhorse requests per server https://dashboards.gitlab.net/d/mO9noY2Wz/increase-canary-traffic?orgId=1&from=now-6h&to=now

From the metrics, we are looking to approximately double the traffic to the current canary stage

changed the description

looks good to me

The path change is complete, we are hovering around 60 req/sec per node where the main stage is seeing about 57 req/sec per node.

closed

marked the checklist item Merge https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1940 to add more GitLab groups to the canary path as completed

marked the checklist item Monitor the canary dashboard and per-node request rates as completed

mentioned in issue #1226 (closed)