Compare Canary and Stable tracks
Description
With the addition of being able to monitor specifically for the two tracks of an app, Canary and Stable in https://gitlab.com/gitlab-org/gitlab-ee/issues/2593, we should also improve our ability to show this data. First, we should make this information available on the environment performance dashboard with the work we are doing in https://gitlab.com/gitlab-org/gitlab-ce/issues/33345 to add support for additional lines series in a single chart. This will allow us to be able to render things like quartiles, but we can also leverage it to overlay the performance of the Canary vs. Stable tracks.
Going further, we should think about how we can calculate the difference in behavior between the two tracks. For example we could leverage the legend data box to provide additional insight. This area is where we show the actual values of the data for the point in time you are currently hovering over. As part of this work we could:
- Add the details for the additional canary line series to the legend
- But we should also provide a percentage comparison relative to the same metric between the
stable
andcanary
tracks.
For example, this may look something like:
- Stable CPU Utilization - Average: 35%
- Canary CPU Utilization - Average: 56% (
+60%
)
Where the percentage difference between the tracks is displayed for easy comparison.
The one area we need to be careful of when trying to calculate the relative difference, is that these values may not be always comparable depending on the metric. For example when dealing with sums across multiple pods like the total error rate across all pods, the canary track will only be getting a small percentage of the traffic. This means the number of requests handled will be smaller, throwing off the comparisons. Perhaps we could flag the metrics that are directly comparable (error percentage, for example) as part of our metric description file, as well as an indicator about which direction is good/bad so we can color the text.
Deliverables:
- Display the canary track, when available, on the same chart as the stable track.
- Display within the legend the additional series, as well as it's data.
- For metric types where it makes sense, provide a calculation of the difference between the tracks.
- Check the design specs below for additional implementation notes
Design
Check the design specs (for implementation notes, spacing, sizes, colors and text copying) — Hide notes in the top-right corner