Skip to content

test(visual): increase failure threshold for charts

Paul Gascou-Vaillancourt requested to merge 2215-charts-failure-threshold into main

What does this MR do?

This attempts to address charts' visual tests flakiness. echarts being somewhat non-deterministic, we sometimes see visual CI jobs fail with very tiny diffs (< 0.03% difference). In this MR, we are applying custom failure threshold to charts stories, so that they don't fail the visual tests when a single pixel is different. We are setting the threshold to an arbitrary 0.03%, which should be high-enough to tolerate recent diffs we've seen, but still low-enough to catch actual breakages.

We are also removing the ability to override those threshold via environment variables. This should not be needed anymore now that visual tests run in containers locally.

Closes #2215 (closed)

Edited by Paul Gascou-Vaillancourt

Merge request reports