Enable puma on two canary nodes in production

Production Change - Criticality 3 C3

Change Objective	Describe the objective of the change
Change Type	ConfigurationChange
Services Impacted	List services
Change Team Members	Name of the engineers involved in the change
Change Severity	C3
Change Reviewer or tested in staging	A colleague who will review the change or evidence the change was tested on staging environment
Dry-run output	If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of the engineer executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )

We are going to resume testing Puma, previously it was disabled due to production incidents related to the database. This change is to re-enable puma on cny-01 and cny-02.

Procedure

• Add node override knife node edit web-cny-01-sv-gprd.c.gitlab-production.internal

    "omnibus-gitlab": {
      "gitlab_rb": {
        "unicorn": {
          "enable": false
        },
        "puma": {
         "enable": false,
         "exporter_enabled": true,
         "exporter_address": "0.0.0.0",
         "listen": "0.0.0.0",
         "worker_processes": 16,
         "worker_timeout": 60,
         "min_threads": 1,
         "max_threads": 2
        }
       }
    },

• Run chef on web-cny-01

• Add node override knife node edit web-cny-02-sv-gprd.c.gitlab-production.internal

    "omnibus-gitlab": {
      "gitlab_rb": {
        "unicorn": {
          "enable": false
        },
        "puma": {
         "enable": false,
         "exporter_enabled": true,
         "exporter_address": "0.0.0.0",
         "listen": "0.0.0.0",
         "worker_processes": 8,
         "worker_timeout": 60,
         "min_threads": 1,
         "max_threads": 4
        }
      }
    },

• Run chef on web-cny-02

Rollback

• Drain the server from the load balancer

/chatops run canary --drain --production

• Remove the node override

added C3 change labels

So that db_pool is set properly we will first need to remove the overrides per environment. This will be done with https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2157

changed the description

marked the checklist item Remove the node override as incomplete

mentioned in merge request gitlab-org/gitlab!19286 (merged)

Puma configured on web-cny-01 on Thu Nov 14 09:33:22 UTC 2019

marked the checklist item Add node override knife node edit web-cny-01-sv-gprd.c.gitlab-production.internal as completed

marked the checklist item Run chef on web-cny-01 as completed

marked the checklist item Add node override knife node edit web-cny-02-sv-gprd.c.gitlab-production.internal as completed

Puma configured on web-cny-02 on Thu Nov 14 09:37:39 UTC 2019

marked the checklist item Run chef on web-cny-02 as completed

workhorse durations https://log.gitlab.net/goto/aa1f2e74f3f1e1244deede66fcf2786a

closed

mentioned in issue #1380 (closed)

mentioned in issue #1381 (closed)