Move Shared Runners fully to GCP

We plan to switch down the DO Shared Runners on GitLab.com and dev.gitlab.org. For this we should:

• check how much will raise the resources usage on existing managers, when all of the CI/CD load will be sent to existing ones in GCP,
• decide if we should add additional managers (currently ~40% of jobs is handled by DO, where we have 4 managers; I suspect we will need to add another managers to GCP to stay stable),
• decide if and how we should update the configuration of managers,
• prepare configuration changes for concurrent/limit/autoscaling settings (if needed),
• apply configuration changes and pause managers in DO,
• update the description of Shared Runners at GitLab.com (currently we mention only DO there, we should replace it with GCP).

TODO for adding new managers:

• create machines:
  • shared-runners-manager-5
    • create /etc/ipaddress.txt with external IP address
  • shared-runners-manager-6
    • create /etc/ipaddress.txt with external IP address
  • gitlab-shared-runners-manager-5
    • create /etc/ipaddress.txt with external IP address
  • gitlab-shared-runners-manager-6
    • create /etc/ipaddress.txt with external IP address
• assign DNS records:
  • shared-runners-manager-5.gitlab.com IN A 35.207.49.66
  • shared-runners-manager-6.gitlab.com IN A 35.207.54.55
  • gitlab-shared-runners-manager-5.gitlab.com IN A 35.207.53.49
  • gitlab-shared-runners-manager-6.gitlab.com IN A 35.207.10.48
• prepare Runner tokens (gitlab-runner register ...)
  • shared-runners-manager-5
  • shared-runners-manager-6
  • gitlab-shared-runners-manager-5
    • for GitLab.com
    • for dev.gitlab.org
  • gitlab-shared-runners-manager-6
    • for GitLab.com
    • for dev.gitlab.org
• prepare new roles in chef:
  • move existing roles:
    • gitlab-runner-srm3 as child of gitlab-runner-srm-gce-us-east1-d
      • move existing secrets from gitlab-runner-srm-gce-us-east1-d to gitlab-runner-srm3
      • switch shared-runners-manager-3.gitlab.com node to gitlab-runner-srm3 role
    • gitlab-runner-srm4 as child of gitlab-runner-srm-gce-us-east1-c
      • move existing secrets from gitlab-runner-srm-gce-us-east1-c to gitlab-runner-srm4
      • switch shared-runners-manager-4.gitlab.com node to gitlab-runner-srm4 role
    • gitlab-runner-gsrm3 as child of gitlab-runner-gsrm-gce-us-east1-d
      • move existing secrets from gitlab-runner-gsrm-gce-us-east1-d to gitlab-runner-gsrm3
      • switch gitlab-shared-runners-manager-3.gitlab.com node to gitlab-runner-gsrm3 role
    • gitlab-runner-gsrm4 as child of gitlab-runner-gsrm-gce-us-east1-c
      • move existing secrets from gitlab-runner-gsrm-gce-us-east1-c to gitlab-runner-gsrm4
      • switch gitlab-shared-runners-manager-4.gitlab.com node to gitlab-runner-gsrm4 role
  • add roles for new managers:
    • gitlab-runner-srm5 as child of gitlab-runner-srm-gce-us-east1-d
      • create secrets for gitlab-runner-srm5 based on gitlab-runner-srm3
      • add Runner token to secrets
    • gitlab-runner-srm6 as child of gitlab-runner-srm-gce-us-east1-c
      • create secrets for gitlab-runner-srm6 based on gitlab-runner-srm4
      • add Runner token to secrets
    • gitlab-runner-gsrm5 as child of gitlab-runner-gsrm-gce-us-east1-d
      • create secrets for gitlab-runner-gsrm5 based on gitlab-runner-gsrm3
      • add Runner tokens to secrets
    • gitlab-runner-gsrm6 as child of gitlab-runner-gsrm-gce-us-east1-c
      • create secrets for gitlab-runner-gsrm6 based on gitlab-runner-gsrm4
      • add Runner tokens to secrets
• bootstrap chef nodes:
  • shared-runners-manager-5, based on gitlab-runner-srm5 role
  • shared-runners-manager-6, based on gitlab-runner-srm6 role
  • gitlab-shared-runners-manager-5, based on gitlab-runner-gsrm5 role
  • gitlab-shared-runners-manager-6, based on gitlab-runner-gsrm6 role
• add new nodes to Prometheus (prometheus-server role):
  • in node job (public_hosts)
  • in shared-runners job (target)
  • in shared-runners-gitlab-org job (target)
• update graph at https://gitlab.com/gitlab-com/runbooks/blob/master/howto/update-gitlab-runner-on-managers.md (runbooks!600)