Skip to content

Draft: Create a way to track disabled tests

Sometimes a test uncovers an issue which will take time to get fixed and so we have to disable that test for the time being. However, we need a way of tracking which tests were disabled so that we can re-enable them later. Ideally, this would need to be in automated fashion as we cannot assume anybody will be running the test manually on a regular basis.

The proposal is to create a scheduled pipeline that would run a single "health check" job on the flaky test. For each disabled test we'd preferably also need a designated "health check" branch. Why a separate repo branch? Well, if we combine multiple test health check jobs on a single branch we'd never know which job actually passed without anybody actively looking. By creating new branches (and removing them once a test is re-enabled) and having notifications set to "Fixed pipeline" we could get an email that a health check job finally passed and we can re-enable the test.

Alternatively, we could have a single branch for test health check jobs and cover scheduled execution of each one of them individually by using CI variables, but it would require more active engagement in terms of checking what health check job actually passed and which test can therefore be re-enabled.

As an example, for the currently non-deterministic 207-disk-media-change test I created a job that installs libvirt and libvirt-perl using upstream RPM pipeline artifacts and builds+installs QEMU from source, then runs the test for 100 times to give the non-determinism enough time to occur. Since jobs run with set -e by default most of the time the job won't need to loop over the whole 100 test runs.

Note that this MR had to be created against the master branch, because no other branch currently exists upstream, but eventually this is supposed to be merged to a different branch.

Merge request reports