Select a testing framework
Select a framework for writing tests. The framework should adhere to the following requirements:
- Be well known (or easy to learn); we want to keep the barrier low for contributors to write tests
- Support writing tests in a BDD style (a BDD-style DSL like mocha)
- Allow tests to be organized by category / context
- Encourage DRY practices (set up / tear down / helpers)
- Support writing browser-based tests when needed. (this may only be needed for the theme)
- Include a description of each test so that the project manager can understand the purpose of the test
- Allow tests to be run individually
- Support generating coverage reports
- Be able to generate HTML-based test and coverage reports that can be reviewed by a project manager
- Output assertion failures in a way that makes it easy to analyze how the actual data differs from the expected data
- Support expect-style assertions
- Not require the number of tests to be maintained manually
Candidates are:
- tap
- mocha + chai
-
jasmine(similar to mocha, but more complex and less pluggable) -
tape(seems to have the same API as tap but with a lot less features and integrations) - jest
I haven't included karma here because it's designed more for browser testing. However, it could be a good solution for testing the UI.
We can consider combining any of these with chai for BDD-style assertions.
Wrap-Up
We decided on mocha + chai + istanbul + sinon. This combination satisfies all the requirements listed above with the least amount of dependencies.
The modular architecture of mocha is very beneficial because it allows us to swap functionality when better options become available. It has very strong support for running individual tests in isolation, which is the most important feature for encouraging contributors to write tests. It also makes daily development more pleasant and efficient.