Identify tests which are ok to fail

Problem to solve

There are some use cases where it's ok for a specific test to fail, but we don't want the pipeline to be stopped because of that failure. For example:

• In test driven development, we might define all the tests up front, and continue to build (and fail) until we've introduced enough code for all the tests to pass. It'd be helpful to mark the tests we know we weren't ready for as "ok to fail".
• A test might be failing for a reason other than the code (eg. external service unavailable), but we still want to complete the pipeline to test the rest of the code
• A failing test might not be relevant to a change that's been made and needs the pipeline to complete (eg. typo on website to push out, but absolute links test is failing)

It is possible today to mark an entire job as "allow_failure", which does not stop the pipeline if the job itself fails (including any test within the job). However, this does not account for the rest of the test suite needing to be run successfully still.

Proposal

Identify and track the results of tests (perhaps just in the "test" job?) and allow users to mark individual tests as "allowed_failure" (quarantined). In subsequent runs, don't fail due to test failures of those marked tests. Allow for removing the flag so that later the test can effect the outcome of the job again.

Links / references

https://confluence.atlassian.com/bamboo/quarantining-failing-tests-289276886.html