All kubernetes resources should have labels matching our Prometheus label taxonomy

At present on GitLab.com, we loosely follow a convention whereby Kubernetes resources (eg, pods, deployments, ingresses, services, etc) have metadata labels matching our Prometheus metadata label taxonomy of type (service identifier), tier, stage (cny, main, blue, green, etc) and shard (marquee, hdd, urgent-cpu-bound etc).

This is being loosely followed, but there are no mechanisms to enforce this, and there has been a drift, which leads to exceptions and complexity in monitoring resources.

Another side effect of this is that we have to configure kube-state-metrics to export all prometheus labels (*). This is expensive and kube-state-metrics specifically warns against doing this. If we moved to a well defined set of labels, we could use these labels instead of *.

Related gitlab-com/runbooks!4335 (merged)

Proposal

  • Add CI jobs to prevent incorrect configuration being deployed.

  • Add Prometheus Alerts (straight to issues) to alert when label requirements are not correctly configured.

Examples of incorrect labelling at present

  1. api nodepools do not have stage labels (see gitlab-com/runbooks!4335 (diffs, comment 839851398))
  2. git nodepools have the incorrect type label (should be git). The current type label should be the shard label. Currently git nodepools do not publish a stage label. See gitlab-com/runbooks!4335 (diffs, comment 839895611)
  3. default and highmem node pools should have stage=main (stage, type, tier, mandatory)
  4. logging pods (deployed via a daemonset) do not have a stage label. Should be stage=main
  5. monitoring service (thanos, prometheus, alertmanager, etc) do not have labels. Should have type, stage, tier etc See gitlab-com/runbooks!4335 (diffs, comment 839923617)
  6. nginx pods, ingress, deployments do not have correct type or stage labels. See gitlab-com/runbooks!4335 (diffs, comment 839926299)
  7. registry nodes do not have stage label: gitlab-com/runbooks!4335 (diffs, comment 839951839)
  8. sidekiq nodes have an incorrect type label consisting of the shard. type should be sidekiq, existing type label should be moved to the shard label. See gitlab-com/runbooks!4335 (diffs, comment 839959454)
  9. web-page nodes should have a stage label: gitlab-com/runbooks!4335 (diffs, comment 840018931)
  10. websocket nodes should have a stage label: gitlab-com/runbooks!4335 (diffs, comment 840027408)
  11. woodhouse resources should be correctly labelled: gitlab-com/runbooks!4335 (diffs, comment 840029422)

Side note: some services don't use stages (eg sidekiq), so why is it important to label them. Having an absent field, instead of a default value for a field makes everything more complicated. For example, when aggregating health across multiple services, how do we handle absent values? Another example are our dashboards, which will automatically include the stage selector in a query. If this is missing from the source data, the graph will not render correctly. Using a default value keeps things simple compared to missing values. On the prometheus side, we follow this policy too: all services have at least a main stage, if not any others.

cc @jarv @hphilipps @skarbek @ggillies

Edited by Andrew Newdigate