Enable GitLab Runner Referees Metrics Feature

Overview

I am trying to enable a new feature for GitLab Runner, called Runner Referees in staging first and then in production. This feature was first implemented in gitlab-org/gitlab-runner!1545 (merged) and gitlab-org/gitlab!20181 (merged). The original issue for this feature is gitlab-org/gitlab#27298 (closed). The first referee we are enabling is the metrics (Prometheus) referee. After this is tested and released, we will create a separate issue to release the network (Suricata) referee.

TODO

• Prepare cookbook update to support the referees configuration entries: gitlab-cookbooks/cookbook-gitlab-runner!26 (merged)
• Ensure that GCP Firewall and host firewalls on CI Prometheus servers allow access from Runner Managers
• Enable it on all staging-shared-runners-manager-X: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/2546
• Add CI Prometheus server in us-east1-b (one of prmX workers was pushed there some time ago and we never added the Prometheus server)
• Enable it on all private-runners-manager-X
• Enable it on all gitlab-shared-runners-manager-X
• Enable it on all shared-runners-manager-X