Skip to content

ci: run visual tests for changed components only

Paul Gascou-Vaillancourt requested to merge 1682-minimal-visual-tests into main

What does this MR do?

This changes the CI setup so that visual tests only run for the components that were actually changed in an MR. This is done by generating a list of changed files basenames and trying to find .stories.js files located in the same directory. The full-blown visual test job can be run manually in an MR and runs as usual in the default branch.

How does it work?

When pushing changes, we get a list of changed files and iterate through it. We try to find any .stories.js file living in the same directory and we run visual regression tests against them. If, for any given file, no .stories.js file can be found, we err on the side of caution and run the full test suite.

Examples

Minimal test run

accordion/
├─ accordion.vue
├─ accordion_item.vue
├─ accordion.stories.js

In this case, changing accordion/accordion_item.vue would result in accordion.stories.js being included in the minimal test run.

Minimal test run (multiple sibling stories)

accordion/
├─ accordion.vue
├─ accordion_item.vue
├─ accordion.stories.js
├─ accordion_item.stories.js

Changing accordion/accordion_item.vue would result in both accordion.stories.js and accordion_item.stories.js being run.

Full test run

accordion/
├─ accordion.vue
├─ accordion.stories.js
accordion_item/
├─ accordion_item.vue

Changing accordion_item/accordion_item.vue would result in the full test suite being run because no .stories.js file can be found next to it.

Note: If accordion_item/ did contain a .stories.js file, only it would be tested in a minimal test run. accordion.stories.js would not be tested. The assumption is that, if a change to accordion_item.vue resulted in visual changes, then its visual test would break and you would have to run update_screenshots which still runs against all stories, which in turn would also update accordion's screenshots if they were affected by the change.

Benefit

The visual CI job is infamously slow. In the last 100 successful pipelines, it took an average of 16 minutes to complete. Considering that we rarely change more than a component at a time, running the full visual test on every commit feels extremely expensive and time-consuming. This is especially true lately, as most MRs only add CSS utils that don't have any impact on the components themselves.

In cases where no component was touched in an MR, the visual_minimal job can complete in around 2mn20s (~85% faster). When a single component was changed, the job could complete in around 4mn40s (~70% faster).

Closes #1682 (closed)

Edited by Paul Gascou-Vaillancourt

Merge request reports