Allow QA tests to be quarantined only in specific environments
Problem
When we quarantine a test it is quarantined everywhere it runs. This is a problem for QA tests when they fail in one environment (e.g., Staging), but pass elsewhere. We lose coverage while waiting for the environment-specific issue to be fixed.
Proposal
Allow tests to be quarantined in a specific environment only (or multiple environments).
We could tag tests using rspec metadata as a hash, e.g., quarantine: 'staging'
or quarantine: ['staging', 'canary', 'production']
. If the metadata is not a hash it would quarantine the test in all environments as it does now.
We'll also need to identify each target execution environment, which might involve CI variables.
Edited by Sanad Liaquat