Prometheus scrape configuration for Kubernetes Pods missing relabel_config for __scheme__

Summary

Our default values.yaml is missing a relabel_config for the kubernetes-pods scrape configuration that will target the Prometheus __scheme__ if the gitlab.com/prometheus_scheme annotation is set.

Without this relabel_config - Prometheus will always fallback to setting to using http for the target metrics URI for the discovered Pod target port regardless of the annotations set on the Pod.

Steps to reproduce

Set gitlab.com/prometheus_scheme: https as an annotation on a Pod set with the gitlab.com/prometheus_scrape: true annotation. Observe that the URI scheme for the Prometheus target URI remains http

Expected behavior

Prometheus should set the __scheme__ to https and apply that to the target URI

Notes

PLEASE NOTE Even with this relabel_config present in the default values - a set of overrides (and duplication of the scrape_config) will be required to set a Prometheus tls_config for the discovered Pod targets.

These overrides will need to include at a minimum a tls_config.server_name setting to specify a name to match the CN or SAN entry in the certificate used to TLS-encrypt the metrics endpoint.

The tls_config may also need to include a ca_file setting if using a certificate signed by a certificate authority not present in the default ca-certificates.crt bundle for the Prometheus image. The values overrides will then also need to include one or more extraSecretMounts: to mount a ca.crt-containing cluster secret into the container for use with the tls_config.ca_file setting.