Make it possible to exclude a test in a particular job (or on review apps)
Problem
We can skip a test if it can't run on a particular environment or pipeline), but in some cases we might need to skip a test in a particular job, e.g., if it runs in multiple jobs in the same pipeline.
For example, qa/specs/features/ee/browser_ui/1_manage/group/restrict_by_ip_address_spec.rb
can't be run against a Review App from CI. We could quarantine it in master pipelines so that it doesn't try to run in review-qa-all
, but then it wouldn't run in the package-and-qa
jobs either. And if we quarantined it it would still try to run in quarantine jobs, but we know it won't work so we want to skip it entirely.
Proposal
Add an exclude
option that allows us to specify the context in which to skip the job. We might have been able to extend the existing skip
rspec tag so that we could specify a hash and have the framework use the hash to determine when to skip the test.
However, since skip
is a built-in rspec feature, it might be simpler to add a new tag. E.g., exclude: { job: 'review-qa-all' }
For example:
it 'is a test', exclude: { job: 'review-qa-all' }
We could also replace the existing :skip_live_env
tag with something similar to the :only
tag. For example, something like exclude: { address: '*.gitlab.com' }