Canary support for custom metrics
We should add first class support for canaries, to custom metrics. Presently you can utilize a query which returns multiple labels to differentiate between stable and canary tracks, but this doesn't take advantage of the additional comparison logic we have between tracks.
To support this, we should add the option to specify a canary specific query, which is similar to how we do it today with the canary logic for CPU/Memory. The reason for entry of a separate query is that there are some metrics, for example the Kubernetes metrics, where it isn't possible to filter between stable and canary tracks with a single label. If we have already run into this situation ourselves, customers will certainly as well.