Implement Salesforce Omniauth integration in gstg and gprd

Production Change - Criticality 3 C3

Change Objective	Implement Salesforce OmniAuth integration
Change Type	Maintenance
Services Impacted	~"Service:Web"
Change Team Members	@cmiskell
Change Severity	C3
Buddy check or tested in staging	Process checked by @jarv ; has also been applied to gstg first
Schedule of the change	Staging: 2:00 2019-05-27, Production: 2019-05-27 22:15
Duration of the change	1hr
Detailed steps for the change.	Described below

Steps:

1. Monitor error rates dashboard (Gitlab Triage: https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&refresh=30s) throughout the process

2. Stop chef on roles:gprd-base-fe-web and roles:gprd-base-fe-api

       knife ssh 'roles:gprd-base-fe-web' 'sudo systemctl stop chef-client
       knife ssh 'roles:gprd-base-fe-api' 'sudo systemctl stop chef-client  

3. Save current, then make change in vault gitlab-omnibus-secrets gprd

       gkms-vault-show gitlab-omnibus-secrets gprd > gitlab-omnibus-secrets.gprd.bak
       gkms-vault-edit gitlab-omnibus-secrets gprd

   Adding the following block (with the actual secret values) to the omniauth_providers array:

       {
           "name": "salesforce",
           "app_id": "SALESFORCE_CLIENT_ID",
           "app_secret": "SALESFORCE_CLIENT_SECRET"
       }
   

4. Sequentially for nodes in roles:gprd-base-fe-web (starting with canary nodes), start/run chef. Example, per node

       knife ssh web-cny-01-sv-gprd.c.gitlab-production.internal "sudo chef-client"
       #For the first 2 or 3, observe logs on the updated node for good looking traffic and no errors in the dashboards, then proceed more rapidly

5. Sequentially in small batches for roles:gprd-base-fe-api, start/run chef (just to avoid causing too many to be restarting at once)

       knife ssh 'roles:gprd-base-fe-api' -C3 "sudo chef-client"

6. Remove backup copy of secrets: rm -f gitlab-omnibus-secrets.gprd.bak

Rollback: Repeat on as many nodes as it has been rolled out to, but removing the omniauth_provider section from gitlab-omniobus-secretes