Skip to content

Allow instantiating SidekiqExporter with settings

Matthias Käppler requested to merge 345802-sk-health-check-settings into master

What does this MR do and why?

We will soon be serving Sidekiq metrics from a separate server process. During this transition, we temporarily plan to instantiate SidekiqExporter twice: once where it is used now (in the worker), and again in the new server process. This is because we want to keep health-checks being served from workers, and only serve metrics from the new server process.

To that end, we need to make a clearer distinction between health-checks and metrics endpoint config, which are currently both part of SidekiqExporter and its settings bucket. This will be a breaking change, so we are taking small incremental steps to remain backwards-compatible.

This MR starts by splitting up settings keys between metrics and health-checks for Sidekiq, so that those servers can be configured independently. For this to remain backwards compatible, we default the new settings keys Settings.monitoring.sidekiq_health_checks to the existing Settings.monitoring.sidekiq_exporter settings, but start using them to configure the server running inside Sidekiq workers.

In parallel, we are emitting these new keys in Omnibus and Helm Charts via separate MRs, but we can ship them independently since they all apply the same fallback logic:

The net result should be: nothing changes, for now. It is merely a refactoring to pave the way forward.

How to set up and validate locally

There are two scenarios we should verify to make sure everything still operates as it currently does.

Scenario 1: New settings are not used; fall back to exporter settings

Verify or set sidekiq_exporter settings:

  1. Edit gitlab.yml and validate or add: 
    development:
  gitlab:
    monitoring:
      sidekiq_exporter:
        enabled: true
        address: 'localhost'
        port: <port>
  2. Note: if <port> is not set, it will default to 8082

Make sure that the first Sidekiq worker sidekiq_0 still emits metrics:

  1. Start Sidekiq, wait for it to boot fully; a server should be listening on <port>
  2. curl localhost:<port>/metrics and verify it is serving Prometheus metrics

Scenario 1: New settings are used; they should be used instead

Verify or set sidekiq_health_checks settings:

  1. Edit gitlab.yml and validate or add: 
    development:
  gitlab:
    monitoring:
      sidekiq_health_checks:
        enabled: true
        address: 'localhost'
        port: <port>
  2. Note: if <port> is not set, it will default to whatever sidekiq_exporter.port is

Make sure that the first Sidekiq worker sidekiq_0 still emits metrics:

  1. Start Sidekiq, wait for it to boot fully; a server should be listening on <port>
  2. curl localhost:<port>/metrics and verify it is serving Prometheus metrics

How to validate in review-app

The server runs on a network internal port, but you can shell into the Sidekiq pod and curl it in the container:

  • open sidekiq-pod and run KUBECTL > Exec > sidekiq
  • replace -- ls with -it -- bash
  • curl endpoint as above 
    $ curl localhost:3807/metrics

Note that in k8s, ports are different from what GitLab (and GDK) use by default.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #345802 (closed)

Edited by Matthias Käppler

Merge request reports