Consistently intermittent flaky e2e tests
Background
The end-to-end tests are annoyingly flaky in general, but workflow_sample_classification.js and workflow_sample_with_cnv_classification.js are particularly so. They will run fine locally or singly, but not pass in CI.
The most frequent pain point is calling checkAlleleClassification
with an analysis, particularly checking analysisComment
. When checked, it will often be an empty string or a partial string (e.g., Expected 'TECHNICA' to equal 'TECHNICAL_ROUND_1'
). Unfortunately, the usual browser.pause
before checking an element has not had any effect, with durations tested up to 3s.
The fact it works when run locally (headless or in a browser) and the occasional partial string suggests a possible buffer or resource issue. However, using the larger runner did not help and I've not been able to find any info on buffer settings in webdriver or jasmine. Running e2e tests individually rather than 2 in parallel seemed to have had an effect for a little while, but that no longer works.
This will eventually no longer be an issue with the new frontend / bdd, but until then we need these tests to actually pass.
Possibilities
- use
waitForExist
/waitForEnabled
to ensure elements have actually loaded before getting their values - Compare state between
checkAlleleClassification
uses that pass and ones that do not - The consistency of where/when it's failing suggests there is an actual underlying issue. Need to determine whether it is in the test framework, host/container set up, actual frontend/backend activity, or something else.
Further investigation has show this particular bout of flakiness may be an indirect result of a malformed IGV config. igv.js repeatedly chain fails (1000s of errors visible in browser console within several seconds), exhausting CPU regardless of available resources.
Changes in !707 (merged) have e2e tests consistently passing when run singly and parallel, so will assume they were in fact the root cause of the e2e flakiness. This time, anyway. Will close along with #2158 (closed) when the MR is merged to dev.