Add prometheus alerts and CI checks to prevent kubernetes resources being applied without appropriate labels
From an alert point of view, we could query various objects exported by Kube-State-Metrics, to figure out which resources are not being correctly annotated.
For example, to find nodes that don't have labels, perhaps we could use something like:
count by (environment, cluster) (node_load1{cluster!=""} unless ignoring(type, tier, stage, shard) node_load1:labeled{cluster!=""} )
If we repeat this type of query for each kubernetes object, we can get lists of unlabeled resources that we can alert on.
cc @alejandro
Edited by Andrew Newdigate