Database indicators: Support for basic auth on the Prometheus API
In !97087 (merged), we introduced a health indicator PatroniApdex
in use for batched background migrations. This indicator connects to the configured Prometheus API to retrieve a apdex metric and this is used to pause batched background migrations if the metric is below a certain threshold.
Configuration from this feature comes from Gitlab::CurrentSettings.current_application_settings.prometheus_alert_db_indicators_settings
, which currently has these properties:
prometheus_api_url
apdex_sli_query
apdex_slo
For GitLab.com, we are migrating from Thanos to Mimir (also see gitlab-com/gl-infra/scalability#3449 (closed)). The Mimir setup requires the use of basic authentication when talking to the (Prometheus-like) API of Mimir.
In order to transition to the Mimir setup, we are going to add support for basic auth when talking to the configured Prometheus API (for the health indicator mentioned above). This should be an optional setting, so we can continue talking to a Prometheus API without basic auth enabled.
Notes on implementation
Option 1: As we're handling a secret here, we can inject this through a ENV variable GITLAB_DATABASE_INDICATORS_PROMETHEUS_USER
and GITLAB_DATABASE_INDICATORS_PROMETHEUS_PASSWORD
(names for illustration). The env variable needs to be specific to "database indicators", as we may need to use other auth settings in other parts of the codebase that also talk to Prometheus. Using an env variable is going to allow us to fetch auth information from vault and expose it to the application.
Option 2: A more flexible implementation would only rely on one env variable GITLAB_DATABASE_INDICATORS_PROMETHEUS_HEADERS
. This would essentially provide a multi-line capable string with relevant HTTP headers. This would allow us to pass in arbitrary headers from configuration, e.g. when moving away from basic auth and using other forms of authentication.
Option 3 (not chosen): Alternatively, we may be able to use an encrypted JSON attribute in ApplicationSetting
to replace prometheus_alert_db_indicators_settings
and add the username/password combination in there. If we choose to go down this route, we can consider a more generic implementation: Specify all relevant HTTP headers in configuration (in this encrypted JSON attribute), which need to be passed when fetching metrics. This will allow us to use basic auth (manually specify Authorization: Basic <base64("$username:$password")>
), along with other headers we may want to specify later. For clarity, this flexibility is currently not required but would lend itself for the implementation, if we choose to use this route.
While Option 3 would work, we also want the ability to use vault to fetch credentials from the runtime environment - which is easiest to do through env variables.
In Migrate Patroni batch migrations to Mimir. (gitlab-com/gl-infra/scalability#3449 - closed) and related discussion gitlab-com/gl-infra/scalability#3168 (comment 1897910059), we tend towards Option 1 and it should be enough to provide the username/password variables as shown above.
Acceptance criteria
The minimal outcome of this issue needs to be:
- Optional ability to use HTTP basic auth when talking to Prometheus in context of database indicators (i.e. here)
- Documentation updated if the configuration is already documented (not sure?)