Validate Mimir tenants in directories and config
Now that we can query Mimir from CI jobs, we can add validation for the directories in mimir-rules/
and list of tenants in gitlab-metrics-config.libsonnet
whether they actually exist as a tenant in Mimir.
Make sure if we validate tenants that exist, we validate tenants with active series. Otherwise we could end up with old tenants that may no longer be writing data.
The list of tenants in Mimir can be retrieved by querying something like:
sum by (user) (cortex_ingester_active_series)
https://dashboards.gitlab.net/goto/bD5v0XYIR?orgId=1
We should also validate the tenants
field in service definition that each element is a valid tenant from gitlab-metrics-config.libsonnet
.
Edited by Gregorius Marco