Deploy GCP managed certs for internal services (non-prod)

Production Change - Criticality 3 C3

Change Objective	Describe the objective of the change
Change Type	ConfigurationChange
Services Impacted	Direct GCP load-balanced services (Monitoring IAPs, Logging IAPs, Camoproxy)
Change Team Members	@bjk-gitlab
Change Criticality	C3
Change Reviewer or tested in staging	@T4cC0re
Dry-run output	If the change is done through a script, it is mandatory to have a dry-run capability in the script, run the change in dry-run mode and output the result
Due Date	Date and time in UTC timezone for the execution of the change, if possible add the local timezone of the engineer executing the change
Time tracking	To estimate and record times associated with changes ( including a possible rollback )

Detailed steps for the change

Due to the slow provisioning step of obtaining the new managed certs, the deployment must be done in two Terraform targeted stages.

• Deploy cert provisioning: tf apply -target=module.${module}.google_compute_managed_ssl_certificate
• Wait for certificate to be provisioned (approx 10 minutes).
• Deploy cert cut-over: tf apply -target=module.${module}.google_compute_target_https_proxy

To check the provisioning status, the gcloud tool can be used:

$ gcloud --project=gitlab-testbed compute ssl-certificates list
NAME                         TYPE          CREATION_TIMESTAMP             EXPIRE_TIME                    MANAGED_STATUS
testbed-monitoring-lb        MANAGED       2020-05-26T06:54:00.591-07:00  2020-08-24T06:04:00.000-07:00  ACTIVE
    prometheus-app.testbed.gitlab.net: ACTIVE
    prometheus.testbed.gitlab.net: ACTIVE

MRs:

• Docs update: gitlab-com/runbooks!2286 (merged)
• Non-Prod Terraform: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1770
• Prod Terraform: https://ops.gitlab.net/gitlab-com/gitlab-com-infrastructure/-/merge_requests/1771

Non-prod Checklist:

• testbed - module=monitoring-lb
• pre - module=monitoring-lb
• gstg - module=monitoring-lb
• gstg - module=camoproxy-lb
• ops - module=monitoring-lb
• ops - module=prod-proxy-iap
• ops - module=nonprod-proxy-iap
• ops - module=proxy-iap
• ops - module=dashboards-internal
• ops - module=dashboards-com-lb
• ops - module=prometheus-dogfood-lb

Prod Checklist: * [ ] gprd - module=monitoring-lb * [ ] gprd - module=camoproxy-lb

Rollback steps

Rollback Terraform MRs and apply. There is no need to wait, as the certificates are already provisioned manually in gcloud.

Changes checklist

• Detailed steps and rollback steps have been filled prior to commencing work
• SRE on-call has been informed prior to change being rolled out
• There are currently no open issues labeled as ServiceMonitoring with severities of ~S1 or ~S2