2022-09-15: Create new GCP Projects for GPU enabled SaaS Linux Runners

Production Change

Change Summary

Create 5 new GCP Projects as part of https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16346.

This CR was composed based on the instructions found in https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt#creating-a-new-environment, and a previous run in #7652 (closed).

Change Details

1. Services Impacted - ServiceCI Runners
2. Change Technician - @rehab
3. Change Reviewer - @rehab
4. Time tracking - 120 minutes
5. Downtime Component - none

Detailed steps for the change

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 60m

• Set label changein-progress /label ~change::in-progress

GCP resources

• 1. Declare an instance of the Google project module 👉 https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4183
• 2. Merge the MR
• 3. run tf init -upgrade and tf plan -out=runners_saas.out > runners_saas_plan in environments/env-projects
• 4. confirm the plan doesn't have any unintended applications.
• 5. run tf apply "runners_saas.out" > applying_plan.
• 6. Service account to be used as a CI variable in order to plan the new project.
  • a. Manually generate a service account key for the terraform-ci service account. https://console.cloud.google.com/iam-admin/serviceaccounts?project=PROJECT_NAME
  • b. add it to 1password.

AWS resources

• 1. create a new IAM user terraform-$ENV with programmatic access.
• 2. attach a new custom policy to the user
  • a. copy policy from existing one.
  • b. name the policy after the new user
  • c. change the path in the second stanza
• 3. create 1password entry "terraform-private/env_vars/$ENV.env" in the production vault
  • a. entry should contain AWS IAM env var declarations for the credentials generated in the previous step.
  • b. Make sure that the access key with special characters can be escaped note: or re-create the key with characters that don't require character escaping or else terraform will not be able to create the bucket.

Follow up steps

• 1. run ./bin/tf-get-secrets and ensure the new env file should is downloaded.

• 3. Create CI env vars for the new environment

  • a. Entries for each var in the private/env-vars file, including the AWS credentials.
  • b. A file entry with key GCLOUD_TERRAFORM_PRIVATE_KEY_JSON whose value is the contents of the private key file created for the terraform-ci user previously.
  • c. Add the environment name to .gitlab/ci/main.jsonnet list, and then run make generate-ci-config and commit the changes.

• Set label changecomplete /label ~change::complete

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 60m

• replace create with delete in the steps above, and execute.

• revert https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/4183 if already merged.

• Set label changeaborted /label ~change::aborted

Monitoring

Key metrics to observe

• Metric: ci-runners:Incident Support:database
  • Location: https://dashboards.gitlab.net/d/ci-runners-incident-database/ci-runners-incident-support-database
  • What changes to this metric should prompt a rollback: A sharp increase in the pending jobs queue, or a sharp decrease in the number of jobs started on runners.

Change Reviewer checklist

C4 C3 C2 C1:

• Check if the following applies:
  • The scheduled day and time of execution of the change is appropriate.
  • The change plan is technically accurate.
  • The change plan includes estimated timing values based on previous testing.
  • The change plan includes a viable rollback plan.
  • The specified metrics/monitoring dashboards provide sufficient visibility for the change.

C2 C1:

• Check if the following applies:
  • The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details).
  • The change plan includes success measures for all steps/milestones during the execution.
  • The change adequately minimizes risk within the environment/service.
  • The performance implications of executing the change are well-understood and documented.
  • The specified metrics/monitoring dashboards provide sufficient visibility for the change.
    • If not, is it possible (or necessary) to make changes to observability platforms for added visibility?
  • The change has a primary and secondary SRE with knowledge of the details available during the change window.
  • The labels blocks deployments and/or blocks feature-flags are applied as necessary

Change Technician checklist

• Check if all items below are complete:
  • The change plan is technically accurate.
  • This Change Issue is linked to the appropriate Issue and/or Epic
  • Change has been tested in staging and results noted in a comment on this issue.
  • A dry-run has been conducted and results noted in a comment on this issue.
  • The change execution window respects the Production Change Lock periods.
  • For C1 and C2 change issues, the change event is added to the GitLab Production calendar.
  • For C1 and C2 change issues, the SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
  • For C1 and C2 change issues, the SRE on-call provided approval with the eoc_approved label on the issue.
  • For C1 and C2 change issues, the Infrastructure Manager provided approval with the manager_approved label on the issue.
  • Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers and this issue and await their acknowledgment.)
  • There are currently no active incidents that are severity1 or severity2
  • If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.