Add pubsubbeat to metrics catalog
Problem
Pubsubbeat is under-represented somewhat on our dashboards. SREs can find it fiddly to make ad-hoc thanos queries to get metrics about pubsubbeat.
Split out of https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11655#note_529168978.
Desired outcome
Add pubsubbeat to metrics catalog, with generated kubernetes dashboards.
Acceptance criteria
-
Pubsubbeat is in the metrics catalog: define metrics for traffic and errors. -
Add "deployment" labels to pubsubbeat pods in order to support the kubeResourcesdirective, that generates kubernetes-oriented dashboards. e.g. https://gitlab.com/gitlab-com/runbooks/-/blob/master/metrics-catalog/services/monitoring.jsonnet#L24 -
Consider tuning the error rate definition to discriminate between non-fatal errors and dropped logs, otherwise we'll be paged spuriously. See https://gitlab.com/gitlab-com/runbooks/blob/master/rules/logging.yml for our current alerts. -
Remove alerts from https://gitlab.com/gitlab-com/runbooks/blob/master/rules/logging.yml if they are now superseded by metrics catalog-generated ones gitlab-com/runbooks!4347 (merged)
Edited by Gonzalo Servat