Add a ScrapeConfig for consul's prometheus job using GCE discovery

Split from #2680 (closed)

As part of our goal to scrape the same Prometheus Jobs that the VMs do from the Prometheus remote agents. On this particular issue will be focus in on the prometheus job consul-discovery

Original configuration in chef for the job, (see chef role https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/gprd-infra-prometheus-server.json)

Original prometheus configuration for consul

- job_name: consul
honor_labels: true
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
enable_http2: true
file_sd_configs:
- files:
- /opt/prometheus/prometheus/inventory/consul.yml
refresh_interval: 5m
- job_name: consul-agent
honor_labels: true
honor_timestamps: true
params:
format:
- prometheus
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /v1/agent/metrics
scheme: http
follow_redirects: true
enable_http2: true
file_sd_configs:
- files:
- /opt/prometheus/prometheus/inventory/consul-agent.yml
refresh_interval: 5m
- job_name: consul-discovery
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
enable_http2: true
relabel_configs:
- source_labels: [__meta_consul_tags]
separator: ;
regex: .*,metrics,.*
replacement: $1
action: keep
- source_labels: [__meta_consul_service]
separator: ;
regex: (blackbox_exporter|prometheus)
replacement: $1
action: keep
- separator: ;
regex: __meta_consul_metadata_(.+)
replacement: $1
action: labelmap
- source_labels: [__meta_consul_service]
separator: ;
regex: (.*)
target_label: job
replacement: $1
action: replace
- source_labels: [__meta_consul_metadata_fqdn, __meta_consul_service_port]
separator: ;
regex: (.+);(.+)
target_label: instance
replacement: ${1}:${2}
action: replace
consul_sd_configs:
- server: 127.0.0.1:8500
tag_separator: ','
scheme: http
allow_stale: true
refresh_interval: 30s
tags:
- metrics
follow_redirects: true
enable_http2: true

This issue will be about add the relevant ScrapeConfig using the GCE discovery (or consul discovery) to scrape the consul metrics across all VM's that aren't kubernetes. We will need to go in details to understand what we try to do above with consul including consul discovery.

At the moment, there is not a clear place to write this scrape configuration, I will follow up with a ticket that will allow us to give home to this and other similar configurations.

You can add the relevant configuration to the helm chart https://gitlab.com/gitlab-com/gl-infra/charts/-/tree/main/gitlab/prometheus-agent