Labeled kube-state-metrics for globally evaluated services

Labeled kube-state-metrics for globally evaluated services

This kube-state-metrics recording rules for globally evaluated services. This will add :labeled version of that that contain our known tier & type labels.

For things like code-suggestions (running in the ai-assist cluster) this will be better than relying on the global type label that is also on the metrics, because it will allow us to separate monitoring infrastructure (prometheus) from the application for saturation metrics. Currently the the metrics from prometheus pods running in ai-assist all have the type="code_suggestions" label, in the :labeled variant this will be overwritten with type="monitoring".

This shows the series that will get a different type label in the :labeled version: https://thanos-query.ops.gitlab.net/graph?g0.expr=count%20by%20(label_type%2C%20type)(kube_pod_labels%7Btype%3D%22code_suggestions%22%7D)&g0.tab=1&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D

For https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23724