Implement CI job to detect invalid test names in ci/.test-failures.${WINDOWSVERSION}.txt files
We need to clean up the .test-failures*
files since some tests no longer exist in the codebase. Ideally, we should add a makefile target and respective CI job to detect this scenario.
A way to detect which test names need to be cleaned out in ci/.test-failures.servercore*.txt
files:
# Massage the testdefinitions.txt file so that it contains only sorted and unique test names
rg --no-filename --no-line-number -oP 'Test[a-zA-Z_0-9]+' (unit|integration)/testsdefinitions.txt | sort | uniq > sortedtestnames.txt
# Massage the ci/.test-failures.servercore1809.txt file so that it contains only top-level test names, sorted and unique
rg --no-line-number -v '/' ci/.test-failures.servercore1809.txt | sort | uniq > .test-failures.servercore1809.uniq.txt
# Output lines in .test-failures.servercore1809.uniq.txt which aren't present in sortedtestnames.txt
$ comm -23 .test-failures.servercore1809.uniq.txt sortedtestnames.txt
TestAuthConfigOverwritingOrder
TestBuildAbort
TestCacheContainerManager_Cleanup
TestCacheContainerManager_CreateCacheContainer
TestCacheContainerManager_FindExistingCacheContainer
TestDefaultManager_Cleanup
TestDefaultManager_ContainerIDs
TestDefaultManager_CreateUserVolumes_CacheVolume_ContainerBased
TestDefaultManager_CreateUserVolumes_CacheVolume_ContainerBased_WithError
TestDockerCommandBuildAbort
TestGetDefaultAuthConfig
TestGetLocalVariableAuthConfig
TestGetRemoteVariableAuthConfig
TestLimits
TestMachineOffPeakIdleLimits
TestNewCacheContainerManager
TestSetupBuildPodServiceCreationError
TestSplitService
TestVolumeMounts
TestVolumes