Skip to content

Support TLS in dedicated metrics exporters

Matthias Käppler requested to merge 364771-exporter-tls into master

What does this MR do and why?

For FIPS compliance, all metrics endpoints scraped by Prometheus need to support TLS. We provide two mechanisms to serve metrics from the Rails monolith:

  1. Rails controller endpoint (/-/metrics - Puma only): This is covered by #353013 (closed) and related MRs.
  2. Dedicated server endpoint (/metrics - Puma and Sidekiq): This MR. This is currently a WEBrick that runs on a separate port. We need to configure it with a certificate and key to enable TLS. We are also working on a replacement system written in Go (gitlab-metrics-exporter) which isn't used yet, but I already added the config mapping, too.

WEBrick supports HTTPS out of the box. We merely need to set the proper SSL config keys. These are mapped from exporter settings in gitlab.yml. See omnibus-gitlab!6155 (merged) for how it's done in Omnibus.

How to set up and validate locally

In gitlab.yml, set the following keys:

monitoring:
  web_exporter: # or sidekiq_exporter, it's the same implementation
    enabled: true # should already be true in development
    tls_enabled: true
    tls_cert_path: /path/to/cert.pem
    tls_key_path: /path/to/key.pem

You could use the test cert and PK from the fixtures folder: https://gitlab.com/gitlab-org/gitlab/blob/72e0329c8d86e804f9ea152b590a73ce591a003e/spec/fixtures/x509_certificate.crt

Then, run Puma (or Sidekiq), and curl the configured address and port, e.g.:

$ curl -v https://localhost:8083/metrics >/dev/null
*   Trying 127.0.0.1:8083...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to localhost (127.0.0.1) port 8083 (#0)
* ALPN, offering http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [10 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1271 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=AU; CN=localhost
*  start date: May 11 14:00:47 2022 GMT
*  expire date: May  8 14:00:47 2032 GMT
*  subjectAltName: host "localhost" matched cert's "localhost"
*  issuer: C=AU; CN=example
*  SSL certificate verify ok.
} [5 bytes data]
> GET /metrics HTTP/1.1
> Host: localhost:8083
> User-Agent: curl/7.79.1-DEV
> Accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4
< Vary: Accept-Encoding
< Server: WEBrick/1.6.1 (Ruby/2.7.5/2021-11-24)
< Date: Wed, 15 Jun 2022 10:48:53 GMT
< Content-Length: 721598
< Connection: Keep-Alive
< 
{ [5 bytes data]
100  704k  100  704k    0     0  7968k      0 --:--:-- --:--:-- --:--:-- 8007k
* Connection #0 to host localhost left intact

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #364771 (closed)

Edited by Matthias Käppler

Merge request reports