Windows test jobs are not running all tests
The Windows unit and integration tests seem to not be running all tests (using ci/.test-failures.servercore1809.txt
as a reference).
As a way to test, download all artifacts from windows 1809 unit tests
and windows 1809 integration tests
jobs, pick a test name from ci/.test-failures.servercore1809.txt
and run the following command (using TestBuildAbort
as an example):
find *.zip -print0 | xargs -0 -n1 unzip -p | rg TestBuildAbort
It should yield results, but in the case of a recent pipeline, it doesn't.
To compare test runs, I've used the following shell command and then pasted the results into a file comparison tool:
find *.zip -print0 | xargs -0 -n1 unzip -p | rg '^=== RUN' | sed 's/=== RUN //g' | grep -v '\/' | sort | uniq | pbcopy
Here is a screenshot of the differences between https://gitlab.com/gitlab-org/gitlab-runner/-/pipelines/321615756/builds and https://gitlab.com/gitlab-org/gitlab-runner/-/pipelines/306686177/builds:
We should also clean up the .test-failures*
files since some tests no longer exist in the codebase. Example: http://gitlab.com/gitlab-org/gitlab-runner/blob/4267601ecd7031be32157fe49c7e6dd49c5703da/ci/.test-failures.servercore1809.txt#L257