Add a scrapeConfig for pushgateway metrics using GCE discovery

Split from #2680 (closed)

As part of our goal to scrape the same Prometheus Jobs that the VMs do from the Prometheus remote agents. On this particular issue will be focus in on the mtail prometheus job

Original configuration in chef for the job, (see chef role https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-infra-prometheus-server.json)

"pushgateway": {
"chef_search": "recipes:gitlab-prometheus\\:\\:pushgateway AND chef_environment:gprd",
"exporter_port": "9091",
"file_inventory": true
},

This issue will be about add the relevant ScrapeConfig using the GCE discovery to scrape the pushgateway metrics across all VM's that aren't kubernetes. Testing the above chef search:

$ knife search node 'recipes:gitlab-prometheus\:\:pushgateway AND chef_environment:gstg'  -i
22 items foundpatroni-registry-v14-02-db-gstg.c.gitlab-staging-1.internal
blackbox-01-inf-gstg.c.gitlab-staging-1.internal
postgres-registry-v14-dr-archive-01-db-gstg.c.gitlab-staging-1.internal
postgres-ci-v14-dr-delayed-01-db-gstg.c.gitlab-staging-1.internal
patroni-ci-v14-104-db-gstg.c.gitlab-staging-1.internal
patroni-registry-v14-01-db-gstg.c.gitlab-staging-1.internal
postgres-main-v14-dr-delayed-01-db-gstg.c.gitlab-staging-1.internal
patroni-main-v14-103-db-gstg.c.gitlab-staging-1.internal
patroni-main-v14-104-db-gstg.c.gitlab-staging-1.internal
patroni-ci-v14-103-db-gstg.c.gitlab-staging-1.internal
patroni-main-v14-101-db-gstg.c.gitlab-staging-1.internal
patroni-ci-v14-101-db-gstg.c.gitlab-staging-1.internal
postgres-registry-v14-dr-delayed-01-db-gstg.c.gitlab-staging-1.internal
postgres-ci-v14-dr-archive-01-db-gstg.c.gitlab-staging-1.internal
patroni-embedding-01-db-gstg.c.gitlab-staging-1.internal
patroni-main-v14-102-db-gstg.c.gitlab-staging-1.internal
patroni-embedding-02-db-gstg.c.gitlab-staging-1.internal
patroni-embedding-03-db-gstg.c.gitlab-staging-1.internal
patroni-ci-v14-102-db-gstg.c.gitlab-staging-1.internal
patroni-registry-v14-03-db-gstg.c.gitlab-staging-1.internal
patroni-registry-v14-04-db-gstg.c.gitlab-staging-1.internal
postgres-main-v14-dr-archive-01-db-gstg.c.gitlab-staging-1.internal

This returns a combination of patroni nodes and a blackbox VM. Not sure if we've a GCE label that maps 1:1 this. cc @stejacks-gitlab

At the moment, there is not a clear place to write this scrape configuration, I will follow up with a ticket that will allow us to give home to this and other similar configurations.

You can add the relevant configuration to the helm chart https://gitlab.com/gitlab-com/gl-infra/charts/-/tree/main/gitlab/prometheus-agent