Add centralized logging to prometheus and grafana hosts
When troubleshooting https://gitlab.com/gitlab-com/infrastructure/issues/4410 it was difficult to determine what was going on because of the amount of logging which was rotated off the host quickly.
We should add all of the Prometheus/Thanos, and Grafana hosts to centralized logging. We should also add some troubleshooting steps into the runbooks
project repository. See https://gitlab.com/gitlab-com/runbooks/blob/master/howto/monitoring-overview.md.
-
Prometheus -
Thanos components (compact, store, query) -
Grafana -
Alertmanager -
Pushgateway