[test] Regression Test Improvements
In the last week we've seen some flakiness in the regression tests due to timeouts on the block creation requests. This will make the tests exit once one is failed in the case of a race, so retrying need not wait on all the tests to finish. Also will fiddle with a few parameters while this is in draft to see if we can speed up the full job and/or reduce flakiness.
Addendum - after some test runs, this makes changes to slow test blocks down by 50%, but increase the parallelism. Additionally it starts the longest running tests first to avoid waiting on the long tail at the end of the run. The end result appears both faster total execution and less flaky.
Merge request reports
Activity
added milestone-planning testing-or-mocknet tooling-or-ci labels
Resolved by Multipartite[Last paragraph of the below as a summary:
By any chance could a
RUN=being set override the exit-on-failure,
such as being able to doRUN=/orRUN=yamlif wanting to check everything,
and (like currently)RUN=refundsto check every test with 'refunds' in the path without stopping early on failure?
]
(Reflecting on this, unit tests (though taking less time total) come to mind; if I remember correctly, all tests in a package complete and have results displayed, but the unit tests don't move onto the next package if there are failures.)
I comment that the
RUN=variable has worked comfortably for me thus far:
First I might run a pipeline (if not actively expecting regression test failures) or else a fullmake test-regressioncommand,
then from the initial summary of which tests failed immediately have a guess about the context
(e.g. if only three .yaml files failed and all with 'refunds' in the path and so suggesting code specificity there,
or if 40-or-so failed suggesting that I've introduced a more basic error somewhere),
then one-by-one withRUN=update a regression test (or/and update main code which affects it)
until satisfied.(For races there's the perspective of being able to know at the start which ones have been able to pass,
thus being able to rule out every race by checking one by one withRUN=,
rather than needing to check all each time since not knowing which ones to check,
possibly failing again before the later tests of interest due to an earlier case already known to be a race,
or else needing to manually go section-by-section withRUN=in order to skip the earlier ones (more tedious than having checking everything once, and/or withTIME_FACTOR=).)I can imagine it could be good for GitLab pipeline resource use to quit as soon as there's any failure,
but upon such a pipeline failure I might still like to have a way to leave a fullmake test-regressionrunning
(while doing something else in that time)
to get a full summary.By any chance could a
RUN=being set override the exit-on-failure,
such as being able to doRUN=/orRUN=yamlif wanting to check everything,
and (like currently)RUN=refundsto check every test with 'refunds' in the path without stopping early on failure?- Last reply by Multipartite
-
Seems the last run of this flaked again - going to revert the parallelism / block speed changes and just keep the change to abort fast and run long tests first for now.
EDIT: Sorry for the thrash, dug a bit deeper into these and will have a more complete fix soon - timeouts due to long invariant checks.
Edited by Ursa (9R) added 5 commits
-
596c7139...9db23552 - 2 commits from branch
develop - 1897e304 - [test] Regression Tests Fail Fast
- 668fc166 - execute longest tests first, only abort early on merge request pipelines
- 26eef094 - couple more tweaks
Toggle commit list-
596c7139...9db23552 - 2 commits from branch
