Add TLS configuration support for Pages metrics endpoint

What does this MR do?

This allows for setting charts configuration for the Pages metrics endpoint

Related issues

Closes #3413 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened

Test plan

See the notes/caveats in the Sidekiq/Webservice metrics merge request test plan

Configuring the included Prometheus chart dependency

See !2671 (merged) for an examples/prometheus/values-tls.yaml set of value overrides - this example presumes:

1. All tls-protected pod endpoints for the scraped metrics will be issued by the same Certificate Authority (CA) - and that it's not included in the base ca-certificates.crt for the Prometheus container image.
2. That CA cert is created as a secret named "metrics.gitlab.tls-ca" ( e.g. kubectl create secret generic --namespace=gitlab metrics.gitlab.tls-ca --from-file=metrics.gitlab.tls-ca=./ca.pem) with a key:value of metrics.gitlab.tls-ca and the CA cert
3. All tls certificates created for the tls-protected pod endpoints will include a SAN of metrics.gitlab

Due to limitations with helm merging multiple values files - all of the chart's default set of serverFiles.prometheus.yml.scrape_configs have to be duplicated in this example overrides. The only difference from the default set of scrape configs is this configuration added to the kubernetes-pods job and kubernetes-service-endpoints job

46a35,37
>           tls_config:
>             ca_file: /etc/ssl/certs/metrics.gitlab.tls-ca
>             server_name: metrics.gitlab

This sets the ca_file to the secret mounted as part of the extraSecretMounts: and defines an arbitrary string to match to a SAN extension entry

Testing

1. Create a certificate for both the sidekiq and webservice metrics endpoints - each with a SAN of metrics.gitlab

   • I used cfssl for my testing, as I had an existing custom CA
   • Example config for my pages metrics cert:

     $ cat csr_pages_metrics.gitlab.json
     {
        "hosts": [
          "metrics.gitlab",
          "pages.metrics.gitlab"
         ],
         "key": {
           "algo": "rsa",
           "size": 2048
         },
         "names": [
           {
             "C": "US",
             "L": "Hillsborough",
             "O": "ROOT ORGANIZATION",
             "OU": "THE ISSUING AUTHORITY",
             "ST": "North Carolina"
           }
         ]
      }

2. Add the certs to Kubernetes (the default name is RELEASE-pages-metrics-tls):

   • kubectl create secret tls --namespace=gitlab gitlab-pages-metrics-tls --cert=./pages.metrics.gitlab.pem --key=./pages.metrics.gitlab-key.pem

3. Add the CA to Kubernetes as metrics.gitlab.tls-ca

   • kubectl create secret generic --namespace=gitlab metrics.gitlab.tls-ca --from-file=metrics.gitlab.tls-ca=./ca.pem

4. Deploy the chart - with the prometheus overrides:

   helm upgrade --install gitlab . --timeout 600s --namespace=gitlab \
   --set global.image.pullPolicy=Always \
   --set certmanager-issuer.email=your_email@gitlab.com \
   --set global.hosts.domain=your_domain \
   --set global.pages.enabled=true \
   --set gitlab.gitlab-pages.metrics.tls.enabled=true \
   -f examples/prometheus/values-tls.yaml --debug

5. You can validate that the endpoints are using TLS using curl from the toolbox container (the CA is not mounted in the toolbox, hence the use of --insecure here):

   • git@gitlab-toolbox-bb756474c-j52nk:/$ curl --insecure --verbose --head https://10.42.2.52:9235/metrics.

      *   Trying 10.42.2.52:9235...
      * Connected to 10.42.2.52 (10.42.2.52) port 9235 (#0)
      * ALPN, offering h2
      * ALPN, offering http/1.1
      * successfully set certificate verify locations:
      *  CAfile: /etc/ssl/certs/ca-certificates.crt
      *  CApath: /etc/ssl/certs
      * TLSv1.3 (OUT), TLS handshake, Client hello (1):
      * TLSv1.3 (IN), TLS handshake, Server hello (2):
      * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
      * TLSv1.3 (IN), TLS handshake, Certificate (11):
      * TLSv1.3 (IN), TLS handshake, CERT verify (15):
      * TLSv1.3 (IN), TLS handshake, Finished (20):
      * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
      * TLSv1.3 (OUT), TLS handshake, Finished (20):
      * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
      * ALPN, server did not agree to a protocol
      * Server certificate:
      *  subject: C=US; ST=North Carolina; L=Hillsborough; O=ROOT ORGANIZATION; OU=THE ISSUING AUTHORITY
      *  start date: Jul 18 20:42:00 2022 GMT
      *  expire date: Jul 18 20:42:00 2023 GMT
      *  issuer: C=US; ST=North Carolina; L=Hillsborough; O=ROOT ORGANIZATION; OU=THE ISSUING AUTHORITY; CN=Jayo Emporium And Fine Certificate Authority
      *  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
      > HEAD /metrics HTTP/1.1
      > Host: 10.42.2.52:9235
      > User-Agent: curl/7.74.0
      > Accept: */*

6. You can also port-forward the Prometheus service - and see the Prometheus targets are scraping https endpoints for the webservice and sidekiq pods: