Consider how to run select orchestrated tests on Staging (and other environments)
Background
We currently run non-orchestrated tests against Staging (and other environments), and we now run all the test scenarios against the same gitlab-omnibus image used to deploy staging.
However, we don't run orchestrated tests against Staging. In some cases that's not possible, because it would require changing the GitLab server configuration. But in other cases GitLab is configured the same way as the orchestrated scenario, so we can run the tests from that scenario. For example, Packages are enabled on Staging, so we can run the packages scenario. The problem is that if we add new package tests they'll fail if they require additional orchestration in a way that's not compatible with how Staging is configured.
Problem
How can we run select orchestrated tests on Staging and other environments, without running others that aren't compatible?
Possible solution
One option is to use supported features RSpec metadata to tag a test that requires a particular feature to be enabled, and set the associated env var to indicate that it's supported in a particular environment.
Issues for exploring staging enablement of existing orchestrated tests
| Rspec meta | SME | Issue to explore enablement on staging |
|---|---|---|
:elasticsearch |
@ebanks |
|
:gitaly_cluster |
@mlapierre, @john.mcdonnell |
|
:repository_storage |
@mlapierre, @john.mcdonnell |
|
:mtls |
@tomi |
|
:mattermost |
@sliaquat |
|
:ldap_tls |
@sliaquat |
NA - LDAP is Self-Managed only feature |
:ldap_no_tls |
@sliaquat |
NA - LDAP is Self-Managed only feature |
:instance_saml |
@sliaquat |
|
:group_saml |
@sliaquat |
|
:smtp |
@sliaquat |
|
:jira |
@sgregory2, @sliaquat |
|
:actioncable |
@dchevalier2 |
|
:registry |
@svistas |
|
:packages |
@svistas |
|
:kubernetes |
@svistas |
|
:object_storage |
@svistas |
|
:geo |
@nwestbury |