Monitor specific tracks to support Canary deployments
With the introduction of canary support on Deploy Boards, we should also add support within Prometheus monitoring. This will also allow us to distinguish between the current then production version of code, from the canary version on a handful of pods.
This is critical to deliver our future vision, of providing feedback on health and determining if the canary is performing better or worse than the current production code.
To power this, we will need a way to differentiate between each track. With our existing Kubernetes label rewriting, we actually capture this today. pod_name
when used with Auto Deploy titles the pod name as environment-canary-id-id
. For example this means that the canary version of production is captured as a new environment production-canary
. This may not be the best solution though, this requires specific pod naming conventions.
Desired path forward:
It doesn't look like the Kubernetes pod labels are being output. In particular, the track
label. This is the standard convention when working with canaries on k8s, and so we should use this method to differentiate.
We should differentiate with track: stable
and track: canary
.