Feedback and next steps on automatic issue creation for test failures
With gitlab-org/gitlab!104983 (merged), we started creating issues automatically for every test that fails in master
.
For now, we're creating the issues in https://gitlab.com/gitlab-org/quality/engineering-productivity/flaky-tests-playground/-/issues and prefix all the labels with wip-
so that people subscribed to actual gitlab-org
labels don't receive notifications yet.
This is because we want to first assess the frequency of new issues created and fine-tune the process before we roll it out in a final issue tracker and with actual gitlab-org
labels.
How to disable the automation (in case anything goes wrong)?
You can easily disable the automation by setting CREATE_ISSUES_FOR_FAILING_TESTS
to false
in https://gitlab.com/gitlab-org/gitlab.
Feedback welcome
This issue exists to gather feedback on this new automation and to discuss improvements and next steps.