Feedback and next steps on automatic issue creation for test failures

With gitlab-org/gitlab!104983 (merged), we started creating issues automatically for every test that fails in master.

For now, we're creating the issues in https://gitlab.com/gitlab-org/quality/engineering-productivity/flaky-tests-playground/-/issues and prefix all the labels with wip- so that people subscribed to actual gitlab-org labels don't receive notifications yet.

This is because we want to first assess the frequency of new issues created and fine-tune the process before we roll it out in a final issue tracker and with actual gitlab-org labels.

How to disable the automation (in case anything goes wrong)?

You can easily disable the automation by setting CREATE_ISSUES_FOR_FAILING_TESTS to false in https://gitlab.com/gitlab-org/gitlab.

Feedback welcome

This issue exists to gather feedback on this new automation and to discuss improvements and next steps.