CI variables are not parameterized for alerting rules
In looking at a few alerts, it looks like we are not properly parameterizing the CI variables that can be used in queries.
For example, here is an example query we use for nginx-ingress throughput:
sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code)
In this case, %{kube_namespace} and %{ci_environment_slug} are variables we are automatically replaced with the corresponding value from CI when viewing an environment's metrics.
However when we configure an alert, we do not parameterize the query and simply pass it along as-is. For example:
sum(rate(nginx_upstream_responses_total{upstream=~"%{kube_namespace}-%{ci_environment_slug}-.*"}[2m])) by (status_code) > 1.0
These should be parameterized, so the alerts work as expected.