[Experiment] Automated QA failure issues creation
With https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/731, we identified a few issues that led to duplicated QA failure issues.
A few improvements have been made:
- gitlab-org/gitlab-qa!574 (merged)
- gitlab-org/gitlab-qa!612 (merged)
- gitlab-org/gitlab-qa!618 (merged)
And we'd like to re-enable the automatic creation of QA failure issues in the canonical project.
First experiment
Scope and config
For now, we'll enable the automatic creation for master
and staging-orchestrated
pipelines only (i.e. https://gitlab.com/gitlab-org/gitlab-qa-mirror and https://ops.gitlab.net/gitlab-org/gitlab-qa), with the default diff ratio threshold of 5%.
Timeframe
As DRI for triaging QA failures during the week of 2020-12-14, I'll be able to monitor the new issues created and tweak the diff ratio threshold to improve the detection if needed.
I'd like to enable the automatic creation at least until the end of my day on 2020-12-17 to have enough data to improve the automation. If it seems to work well, we could leave it enabled, and even enable it for other environments.
Second experiment
Scope and config
For now, we'll enable the automatic creation for master
and staging-orchestrated
pipelines only (i.e. https://gitlab.com/gitlab-org/gitlab-qa-mirror and https://ops.gitlab.net/gitlab-org/gitlab-qa), with a custom diff ratio threshold of 15%.
Timeframe
We'll run this experiment from 2021-01-06 to 2021-01-07.
Resources
How to disable the automatic creation of issues?
Simply set the DISABLE_RELATING_FAILURE_ISSUES
project variable (in gitlab-org/gitlab-qa-mirror
) with any value (e.g. true
).
How to tweak the diff ratio threshold?
Simply set the QA_FAILURES_MAX_DIFF_RATIO
project variable (in gitlab-org/gitlab-qa-mirror
) with a percentage value (e.g. 0.10
for 10%).
relate_test_failures
job in dry-run mode?
How to run the Simply set the QA_FAILURES_REPORTER_OPTIONS
project variable (in gitlab-org/gitlab-qa-mirror
) with --dry-run
.