Enhance Prometheus environment detection
In Kubernetes, it can be common to only have a single Ingress across multiple applications within a single cluster. When this occurs, we can't expect a CI_ENVIRONMENT_SLUG
variable to be present on the Ingress as it supports multiple environments.
In cases like these, where a single metrics source could provide metrics across a variety of environments we could look for alternative identifiers as well. For NGINX (the most common k8s ingress), we have two options:
Upstream metrics
Use upstream
metrics, which have an upstream
label. These metrics are shown whenever NGINX is reverse proxying data to an upstream host. The upstream
label then seems to be of the form namespace-service-port
. For example:
upstream="default-java-frontend-production-5000"
upstream="java-frontend-6-production-5000"
upstream="gitlab-gitlab-9090"
upstream="gitlab-gitlab-8065"
upstream="gitlab-gitlab-9090"
upstream="rubyprj-1-production-5000"
upstream="rubyprj-1-staging-5000"
...
Standard metrics
Use the incoming traffic metrics, which denote what hostname the requests are being received for. Note that these metrics do not support latency, which are only available on the upstream metric sets. For example:
server_zone="rubyprj.35.185.40.222.nip.io"
server_zone="rubyprj-staging.35.185.40.222.nip.io"
server_zone="gitlab.35.185.40.222.nip.io"
...
If you would like to take a closer look yourself, use this Prometheus server: https://prometheus.35.185.40.222.nip.io
I prefer number one, since it includes latency. If we could detect something of the form, it should work:
KUBE_NAMESPACE-
CI_ENVIRONMENT_SLUG-dddddd