[Hosted Runners VPC] Scale up the private shard

• Create a GCP project
  • Merge https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9650.
  • Merge https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9651.
  • Merge https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9652.
  • Request quota increase in GCP.
  • add new projects to the CI Project Cleaner.
    • Append the GCP project ID to gcp_projects https://ops.gitlab.net/gitlab-com/gl-infra/ci-project-cleaner/-/merge_requests/21
    • Add the GCP project ID to vault-production GCP projects. 👉 https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9671
    • Add the GCP project ID to infra-mgmt impersonated accounts. 👉 https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/1263
• Create Chef roles {chef-repo MR} (ensure concurrent = 0) 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/5280+.
• Create runner-manager VMs 👉 https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/9685.
• Register new runner managers and add tokens to Vault (6 tokens for each VM, that is 6x4 = 24 - automate?)
  • gitlab.com/gitlab-org group runner
  • gitlab.com/gitlab-com group runner
  • gitlab.com/charts group runner
  • dev.gitlab.org instance runner
  • ops.gitlab.net instance runner
  • staging.gitlab.com instance runner
• Set concurrent = 1 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/5293
• Run deployer & verify runner-manager can create VMs.
• Increase concurrent = 100 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/5294
• Verify jobs are succeeding 👉 #3865 (comment 2211210921)
• Increase concurrent = 1200 👉 https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/5295+s
• Inform EOC of the change