Move prometheus based alerts to mimir
This is blocked by #2684 (closed)
The following hand-crafted files have no recording rules and are just alerts.
They should be briefly audited to confirm that they have adequate scope in their queries and they are still relevant. Mimirtool can apparently do this, but git grep in runbooks can as well.
Then they should be moved to a new alerts directory in mimir.
-
airflow.yml -
alertmanager.yml -
chefs.yml -
cloud_nat.yml gitlab-com/runbooks!6957 (merged) -
cloud_sql.yml -
cloudflare.yml gitlab-com/runbooks!6958 (merged) -
commit-workers.yml gitlab-com/runbooks!6959 (merged) -
container.yml gitlab-com/runbooks!6957 (merged) -
decomposed-database.yml -
elastic-clusters.yml #2941 -
external-dns.yml gitlab-com/runbooks!7051 (merged) -
gitlab-com-latencys.yml gitlab-com/runbooks!7052 (merged) -
gitlab-com-repositories.yml gitlab-com/runbooks!7052 (merged) -
gitlab-com-search.yml gitlab-com/runbooks!7058 (merged) -
gitlab_job.yml gitlab-com/runbooks!6957 (merged) -
kubernetes-horizontalpodautoscaler.yml gitlab-com/runbooks!6957 (merged) -
kubernetes-resources.yml gitlab-com/runbooks!6957 (merged) -
kubernetes-storage.yml gitlab-com/runbooks!6957 (merged) -
kubernetes-system.yml gitlab-com/runbooks!6957 (merged) -
kubernetes.yml gitlab-com/runbooks!7059 (merged) -
logging.yml #2941 -
mailroom.yml gitlab-com/runbooks!6957 (merged) -
omnibus.yml gitlab-com/runbooks!6957 (merged) -
osquery.yml gitlab-com/runbooks!6957 (merged) -
pages-gitlab-io-status.yml gitlab-com/runbooks!7060 (merged) -
patroni.yml gitlab-com/runbooks!7088 (merged) -
pgbouncer.yml gitlab-com/runbooks!7089 (merged) -
praefect.yml gitlab-com/runbooks!7087 (merged) -
prometheus-operator.yml gitlab-com/runbooks!6957 (merged) -
pull-mirror-queues.yml gitlab-com/runbooks!6957 (merged) -
registry-db.yml gitlab-com/runbooks!7061 (merged) -
registry-gc-queues.yml gitlab-com/runbooks!7172 (merged) -
remote-mirrors.yml gitlab-com/runbooks!6957 (merged) -
sidekiq-queue-latency.yml gitlab-com/runbooks!6957 (merged) -
sidekiq-queues.yml gitlab-com/runbooks!7161 (merged)
These alerts used to be deployed to all Prometheus instances, meaning that on some prometheus instances they wouldn't actually record anything as the metrics aren't available on that deployment.
For Mimir, we should use the scoped directory structure of the format '%(tenantName)s/%(envName)s/%(serviceName)s/%(baseName)s.yml'
. We should make sure that we move the handcrafted files to the right directory: in the tenant that contains the metrics, and for the service that needs these alerts. If a alerts from a certain file need to live in multiple directories because they apply to multiple services, the we should probably migrate these to jsonnet so we can reuse them.