Trigger full suite of QA tests when a feature flag is changed
Summary
We send a Slack notification to a QA channel when a feature flag is changed. However, to run the QA tests we rely on scheduled pipelines, or ones that are triggered by a deployment. That may be sufficient in some cases, but it could result in a substantial delay before QA test results are generated and reviewed. Especially since pipelines that include the full suite run less frequently than smoke tests, and the changes behind the feature flag might not be covered by smoke tests.
Inspired by gitlab-com/gl-infra/production#5367 (closed)
We may need to limit how frequently a pipeline can be triggered that way, to avoid starting multiple pipelines in quick succession. For example if someone enables a flag and then quickly disables it after a manual test reveals a problem.
Proposal
-
Trigger the full suite of QA tests against the relevant environment when a feature flag is changed and include the pipeline URL in the message sent to the relevant QA channel => MR -
Send chat service's user id as CHAT_USER_ID
to triggered pipelines so that it can be later used to@mention
the user in case of a failure => MR -
For bonus points: when the QA pipeline is triggered, send the feature flag with the trigger so that the test session report can include it to make it clear that that pipeline was triggered in response to a feature flag change. -
Do not trigger the pipeline when a feature flag is toggled for a specific group, project or a user except for the gitlab-qa-sandbox-group
and thegitlab-qa
user. => MR -
@mention
the user on Slack when an end-to-end test triggered due to their FF toggle fails. => MR -
Enable TRIGGER_E2E_TESTS
env variable on the ChatOps project.
- [ ] For extra bonus points: make it possible to trigger a defined action to verify a feature flag change, where the first action defined is to trigger the QA tests. Another action could be to make an API call and return the results. Put on the backlog because it's not clear if it's required: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1063