Create generic test runner
Having a generic runner would provide a framework to allow users to add tests to the suite as needed.
If there's another out there that can meet the desired needs as shown below, then look at those options, but so far none have been found so likely need to build a simplified test runner. Features should include:
- tests are of type pass/fail as well as complete/incomplete
- failed/incomplete tests should return a non-zero exit code
- both types should have the equivalent of "passed with warnings" with exit code 0
- test failure does not include a stack trace (since the test code is not relevant)
- tests should have the ability to aggregate supplemental data (arbitrary as returned by the test)
- test framework provides reporting capability in multiple possible formats (e.g. html, json, console) - see #25 (closed)
- test runner should expose a test function
- provides test name, type, and function to execute
- optionally can provide a config setting parameter to enable/disable a test (i.e. string with name of setting property, or boolean to always/never run)
- test function passes execution context (page, browser console log, test config settings) to the supplied function
- supplied function returns either:
- object with
result
and optionaldata
property - to support the use of an assertion library (e.g.
chai
)- nothing, which is assumed to mean the test is successful (passed/completed)
- throws an error, which is assumed to mean the test is unsuccessful (failed/not completed)
- object with
- if supplied function is async or returns a Promise, await the results (functionally handled by always awaiting the results the function, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await)
- test function provides logging/reporting
- test framework aggregates results from the test functions (per test, and a summary)
- provides test name, type, and function to execute
Some of this is already covered by existing capabilities from #43 (closed).