Test definitions is not picking up Windows specific tests
Overview
To run our tests in parallel we have a job test_defintions that will create a list of tests that need to be run. We are running this inside of a Linux environment so any test that is Windows-specific never runs so we are never testing windows specific tets, since they are missing from the testdefintions.txt
artifact.
For example, this pipeline creates the following testsdefinitions.txt. We also have tests that can only run on Windows (since they are testing windows specific code paths) like register_windows_test.go. If you look at the generated testdefitions both TestRegisterDefaultWindowsDockerCacheVolume
& TestDefaultWindowsShell
are missing, this is because we are generating that file inside of a Linux environment so they are not considered.
In total right now we have 8 windows speicfic tests, the others all work for unix so they are being tested. But this number will grow in the future.
The windows specific tests are listed below:
- https://gitlab.com/gitlab-org/gitlab-runner/blob/9b022da95c3fe5c920759986445522278a0ba137/commands/register_windows_test.go
- https://gitlab.com/gitlab-org/gitlab-runner/blob/9b022da95c3fe5c920759986445522278a0ba137/executors/docker/internal/volumes/parser/windows_parser_test.go
- https://gitlab.com/gitlab-org/gitlab-runner/blob/9b022da95c3fe5c920759986445522278a0ba137/helpers/path/windows_path_test.go
- https://gitlab.com/gitlab-org/gitlab-runner/blob/9b022da95c3fe5c920759986445522278a0ba137/executors/docker/internal/volumes/utils_windows_test.go
Proposal
Generate two different testdefitions for both Linux and Windows for example testdefitions-unix.txt
& testdefitions-windows.txt
and have each job pick the right one.