Skip to content

Windows test jobs are not running all tests

The Windows unit and integration tests seem to not be running all tests (using ci/.test-failures.servercore1809.txt as a reference).

As a way to test, download all artifacts from windows 1809 unit tests and windows 1809 integration tests jobs, pick a test name from ci/.test-failures.servercore1809.txt and run the following command (using TestBuildAbort as an example):

find *.zip -print0 | xargs -0 -n1 unzip -p | rg TestBuildAbort

It should yield results, but in the case of a recent pipeline, it doesn't.

To compare test runs, I've used the following shell command and then pasted the results into a file comparison tool:

find *.zip -print0 | xargs -0 -n1 unzip -p | rg '^=== RUN' | sed 's/=== RUN   //g' | grep -v '\/' | sort | uniq | pbcopy

Here is a screenshot of the differences between https://gitlab.com/gitlab-org/gitlab-runner/-/pipelines/321615756/builds and https://gitlab.com/gitlab-org/gitlab-runner/-/pipelines/306686177/builds:

image

We should also clean up the .test-failures* files since some tests no longer exist in the codebase. Example: http://gitlab.com/gitlab-org/gitlab-runner/blob/4267601ecd7031be32157fe49c7e6dd49c5703da/ci/.test-failures.servercore1809.txt#L257