Skip to content

Fail CI build if test failures not updated

Pedro Pombeiro (OOO from July 16th till Aug 7th) requested to merge pedropombeiro/27999-fail-build-if-test-failures-not-updated into main

What does this MR do?

It adds a check_windows_test_ignore_list makefile target and check windows 1809 test ignore list/check windows 2004 test ignore list CI jobs.

It also removes names of tests that not longer exist from the list, as well as those of tests that are no longer failing.

Why was this MR needed?

Whenever a developer deleted a test from the codebase, and that test was ignored in the .test-failures.servercore1809.txt there was no feedback that that entry in the text file had been orphaned. With this CI job, the developer will get warned by a failing job.

What's the best way to test this MR?

Failing job: https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/1358910039

To see a failure: reset the branch to a point where we haven't yet touched ci/.test-failures.servercore1809.txt.

In the project root on the local development environment:

$ mkdir -p unit integration && \
  make parallel_test_prepare TESTSDEFINITIONSPATH=integration/testsdefinitions.txt TESTFLAGS='-cover -tags=integration' && \
  make parallel_test_prepare TESTSDEFINITIONSPATH=unit/testsdefinitions.txt TESTFLAGS='-cover' && \
  docker run -ti -v $PWD:/src -w /src registry.gitlab.com/gitlab-org/gitlab-runner/ci:1.13.8-12 bash
# Inside the docker container
root@81f05e87f3e4:/src# make check_windows_test_ignore_list
# Checking for orphaned test names in ignore file

=========================================================================================================================
Please remove the following test names from ci/.test-failures.servercore1809.txt. They no longer exist in the codebase:

TestAuthConfigOverwritingOrder
TestBuildAbort
TestCacheContainerManager_Cleanup
TestCacheContainerManager_CreateCacheContainer
TestCacheContainerManager_FindExistingCacheContainer
TestDefaultManager_Cleanup
TestDefaultManager_ContainerIDs
TestDefaultManager_CreateUserVolumes_CacheVolume_ContainerBased
TestDefaultManager_CreateUserVolumes_CacheVolume_ContainerBased_WithError
TestDockerCommandBuildAbort
TestGetDefaultAuthConfig
TestGetLocalVariableAuthConfig
TestGetRemoteVariableAuthConfig
TestLimits
TestMachineOffPeakIdleLimits
TestNewCacheContainerManager
TestSetupBuildPodServiceCreationError
TestSplitService
TestVolumeMounts
TestVolumes
=========================================================================================================================

make: *** [Makefile:173: check_windows_test_ignore_list] Error 1

If you reset the branch to the latest commit and re-run the procedure, no error should be displayed this time.

What are the relevant issue numbers?

Closes #27999 (closed)

Edited by Pedro Pombeiro (OOO from July 16th till Aug 7th)

Merge request reports