Prometheus scrape configs in HA setup may conflict
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/40949 and gitlab-com/support/support-team-meta#1513 (closed)
In a scaled or high availability configuration, scrape configs must be created manually. The manual configurations can conflict with some of the built-in scrape configs that are automatically enabled on the Prometheus server itself.
Given the following gitlab.rb
on the Prometheus node:
nginx['enable'] = true
node_exporter['enable'] = true
alertmanager['enable'] = false
gitaly['enable'] = false
gitlab_monitor['enable'] = false
gitlab_workhorse['enable'] = false
postgres_exporter['enable'] = false
postgresql['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
external_url 'http://grafana.example.com'
grafana['enable'] = true
grafana['admin_password'] = 'secret'
prometheus['listen_address'] = '0.0.0.0:9090'
prometheus['scrape_configs'] = [
{"job_name"=>"nginx", "static_configs"=>[{"targets"=>["68.183.126.89:8060","104.248.120.116:8060","142.93.121.145:8060","104.248.52.188:8060"]}]},
{"job_name"=>"node", "static_configs"=>[{"targets"=>["68.183.126.89:9100","104.248.120.116:9100","142.93.121.145:9100","104.248.52.188:9100"]}]},
...
Nginx is enabled so we can access Grafana and Node Exporter is enabled so we can get statistics on the Prometheus server itself. This automatically produces scrape config entries with job names 'nginx' and 'node', which conflict with the 'nginx' and 'node' job names we created manually for our other nodes. When this happens, Prometheus will not start.
We could arbitrarily change the job names of our manual entries but we want them to be consistent with defaults so Grafana dashboards work properly.
An easy fix may be to add a configuration option that turns off collection of the automatic scrape configs. Users can then manually add Nginx and Node Exporter to their respective jobs alongside other nodes.