Generated prometheus.yml does not work by default

Overview

It took some figuring out to get prometheus working in GDK. There are two problems that I found:

• Prometheus is run in docker, and this cannot connect to gdk.test as configured on OSX https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/index.md#set-up-gdktest-hostname - I found that I needed to use host.docker.internal as specified in the docker-for-mac docs https://docs.docker.com/docker-for-mac/networking/
• The metrics path for praefect/gitaly is wrong. It needs to specify metrics_path: '/metrics'

For future reference, it was super useful finding http://localhost:9090/targets to debug this.

gdk.yml:

hostname: gdk.test

prometheus:
  enabled: true

Generated prometheus.yml:

global:
  # By default, scrape targets every 15 seconds.
  scrape_interval: 15s

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'gdk-monitor'

# A scrape configuration containing four endpoints to scrape. Requires that
# `gdk.test` is configured. See doc/index.md#set-up-gdktest-hostname for more
# information.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped
  # from this configuration.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['gdk.test:9090']
  - job_name: 'gitlab'
    scrape_interval: 30s
    metrics_path: '/-/metrics'
    static_configs:
      - targets: ['gdk.test:3000']
  - job_name: 'gitaly'
    scrape_interval: 5s
    static_configs:
      - targets: ['gdk.test:9236']
  - job_name: 'praefect'
    scrape_interval: 5s
    static_configs:
      - targets: ['gdk.test:10101']
  - job_name: 'gitlab-sidekiq'
    scrape_interval: 30s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['gdk.test:3807']

Modified "fixed" prometheus.yml:

global:
  # By default, scrape targets every 15 seconds.
  scrape_interval: 15s

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'gdk-monitor'

# A scrape configuration containing four endpoints to scrape. Requires that
# `host.docker.internal` is configured. See doc/index.md#set-up-gdktest-hostname for more
# information.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped
  # from this configuration.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['host.docker.internal:9090']
  - job_name: 'gitlab'
    scrape_interval: 30s
    metrics_path: '/-/metrics'
    static_configs:
      - targets: ['host.docker.internal:3000']
  - job_name: 'gitaly'
    scrape_interval: 5s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['host.docker.internal:9236']
  - job_name: 'praefect'
    scrape_interval: 5s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['host.docker.internal:10101']
  - job_name: 'gitlab-sidekiq'
    scrape_interval: 30s
    metrics_path: '/metrics'
    static_configs:
      - targets: ['host.docker.internal:3807']

Proposal (optional)

Environment (optional)

• Operating System: Windows/macOS/Linux
• The contents of your gdk.yml
• Ruby version: <!-- output of ruby --version -->
• GDK version: <!-- output of git rev-parse --short HEAD -->