Explore Options for Running Full QA Suite on Staging More Frequently
The full QA suite against Staging currently only runs after post-deploy migrations now. Before, we used to have the full suite run after a feature flag toggle for the environment, but this has since been replaced with only the sanity suite. As a result, the full QA suite runs very sparingly now against Staging.
For example, during this week between 2023-07-10
and 2023-07-14
, the full suite only ran once during the following days:
-
2023-07-11
https://ops.gitlab.net/gitlab-org/quality/staging/-/pipelines/2103932 -
2023-07-12
https://ops.gitlab.net/gitlab-org/quality/staging/-/pipelines/2107758 -
2023-07-14
https://ops.gitlab.net/gitlab-org/quality/staging/-/pipelines/2113953
Several E2E tests can only run against Staging which are non-reliable tests. Examples of such tests can be found across various stages and groups, such as Fulfillment, Package and Secure, and we are losing valuable coverage from these areas. Some of these tests cannot be promoted to reliable due to relying on integrations which are unreliable and out of our control (ex: Fulfillment E2E tests which require integration with Zuora)
Potential Solutions
- Introduce full scheduled runs again
- Re-implement running the full QA suite after a feature flag toggle
- Run full QA suite after Staging Canary deployments (similar to what we do today for Production Canary)
- Include running only the subset of tests labeled as
only: { pipeline: [:staging, :staging-canary] }
(can apply to feature flag toggle, staging canary deployments, or both)