Testing Strategy: Mutation testing
Problem
While working on #353293 (closed), a number of issues 1 were found by manual testing or very high level automated tests, which indicate missing low level test coverage.
Proposal
Use mutation testing to proactively find testing gaps. Mutation testing works by altering ("mutating") the application code and then running the tests to see if they fail with the altered code.
Python
There are a number of Python modules for doing mutation testing. This article provides a high-level summary of them (though it lists mutatest as unmaintained, which does not appear to be entirely accurate).
C#
Mutation testing is not as mature in C#; Stryker seems to be the only game in town.
Considerations
- Mutation testing can take a long time, since it has to frequently recompile the application code and re-execute the tests. The longer the code takes to compile and the tests take to run, the more this factor is exacerbated.
- Although it is generally better at finding test gaps than traditional test coverage, it is definitely not perfect. In particular, since it can only mutate code that already exists, it is not necessarily good at finding missing test cases for scenarios that the existing code does not cover at all.
Alternatives
Traditionally test gaps have been identified via test coverage analysis; however, such analysis is limited:
Traditional test coverage (i.e line, statement, branch, etc.) measures only which code is executed by your tests. It does not check that your tests are actually able to detect faults in the executed code.
https://pitest.org/
This is especially true when a test suite has a high proportion of high level tests. Such tests are likely to "cover" a large amount of code, but may not have the breadth or depth of assertions needed to find faults with the covered code.
However, if mutation testing is prohibitively expensive to implement or run continuously, falling back on test coverage analysis might still have value for finding completely uncovered code.