Migrate Patroni batch migrations to Mimir.

This will be the first of the fast follow issues once &1107 is completed.

We had a great deal of conversation about the right way to handle this in this issue -- #3168 (comment 1897910059) -- and I am saving the relevant pieces here.

gitlab-org/gitlab!97087 (diffs) is the MR that created this feature in the first place.

GitLab-gstg needs access to the Gitlab - gstg tenant, while GitLab-gprd needs access to the Gitlab - gprd tenant. The configuration is different per environment:

[ gstg ] production> Gitlab::CurrentSettings.current_application_settings.prometheus_alert_db_indicators_settings
=> 
{"apdex_slo"=>{"ci"=>0.997, "main"=>0.997, "main_cell"=>0.997},
 "wal_rate_slo"=>{"ci"=>70000000, "main"=>70000000, "main_cell"=>70000000},
 "apdex_sli_query"=>
  {"ci"=>
    "avg_over_time(gitlab_service_apdex:ratio_5m{env='gstg',environment='gstg',monitor='global',type='patroni-ci'}[5m])",
   "main"=>
    "avg_over_time(gitlab_service_apdex:ratio_5m{env='gstg',environment='gstg',monitor='global',type='patroni'}[5m])",
   "main_cell"=>
    "avg_over_time(gitlab_service_apdex:ratio_5m{env='gstg',environment='gstg',monitor='global',type='patroni'}[5m])"},
 "prometheus_api_url"=>
  "http://thanos-query-frontend-internal.ops.gke.gitlab.net:9090",
 "wal_rate_sli_query"=>
  {"ci"=>
    "avg(postgres:pg_xlog_bytes_per_second:rate5m{env='gstg', type='patroni-ci'})",
   "main"=>
    "avg(postgres:pg_xlog_bytes_per_second:rate5m{env='gstg', type='patroni'})",
   "main_cell"=>
    "avg(postgres:pg_xlog_bytes_per_second:rate5m{env='gstg', type='patroni'})"}}

@reprazent suggests "I don't think the configuration of the authorization should come in through the GitLab's AppliationSetting database record though, but rather injected into the env from vault.

I'm not worried about consuming the environment to add basic auth to these requests, I don't know what it would take to get the value from vault into the env, but I assume that there's enough examples there. The secrets we need to use already exist in vault (link to vault)."

@abrandl said:

"Effort wise, we'd ideally put this into a gitlab-org/gitlab issue for visibility, extend JSON configuration to support basic auth and arbitrary headers, teach generic code to use auth and headers, teach the database health indicators parts to configure the client to use auth and headers, get it tested, add documentation and get it deployed, plus changing the runtime configuration in relevant environments."

We have moved this into a fast follow issue after the Mimir migration.

Edited by Stephanie Jackson