7. Create runner-manager VMs for XL linux with chef-client

• Create the new definitions in TF 👉 TBD
• ssh into each of the new VMs, example:

  gcloud compute os-login ssh-keys add --key-file path/to/your/public/key --ttl=0
  scp runners-manager-shared-green-2.c.gitlab-ci-155816.internal:oslogin_restore.sh .
  scp -i /home/rehab/.ssh/google_compute_engine oslogin_restore.sh rhassanein_gitlab_com@runners-manager-saas-linux-medium-amd64-gpu-standard-blue-1.c.gitlab-ci-155816.internal:
  ssh -i path/to/your/private/key rhassanein_gitlab_com@runners-manager-saas-linux-medium-amd64-gpu-standard-blue-1
  sudo su
  vim /etc/nsswitch.conf #remove all occurance of `cache_oslogin` and `oslogin`
  chef-client
  vim /etc/nsswitch.conf #restore to previous values

• Ensure gitlab-runner service is disabled by default on all the VMs.
• Verify chef-client executes successfully.
• Verify config.toml have the correct configs.
• Disable chef-client.
• Ensure gitlab-runner service is disabled by default on all the VMs.
• Set concurrent to 0 in config.toml.

reference issue: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23766