Skip to content

[Experiment] Automated QA failure issues creation

With https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/731, we identified a few issues that led to duplicated QA failure issues.

A few improvements have been made:

And we'd like to re-enable the automatic creation of QA failure issues in the canonical project.

First experiment

Scope and config

For now, we'll enable the automatic creation for master and staging-orchestrated pipelines only (i.e. https://gitlab.com/gitlab-org/gitlab-qa-mirror and https://ops.gitlab.net/gitlab-org/gitlab-qa), with the default diff ratio threshold of 5%.

Timeframe

As DRI for triaging QA failures during the week of 2020-12-14, I'll be able to monitor the new issues created and tweak the diff ratio threshold to improve the detection if needed.

I'd like to enable the automatic creation at least until the end of my day on 2020-12-17 to have enough data to improve the automation. If it seems to work well, we could leave it enabled, and even enable it for other environments.

Second experiment

Scope and config

For now, we'll enable the automatic creation for master and staging-orchestrated pipelines only (i.e. https://gitlab.com/gitlab-org/gitlab-qa-mirror and https://ops.gitlab.net/gitlab-org/gitlab-qa), with a custom diff ratio threshold of 15%.

Timeframe

We'll run this experiment from 2021-01-06 to 2021-01-07.

Resources

How to disable the automatic creation of issues?

Simply set the DISABLE_RELATING_FAILURE_ISSUES project variable (in gitlab-org/gitlab-qa-mirror) with any value (e.g. true).

How to tweak the diff ratio threshold?

Simply set the QA_FAILURES_MAX_DIFF_RATIO project variable (in gitlab-org/gitlab-qa-mirror) with a percentage value (e.g. 0.10 for 10%).

How to run the relate_test_failures job in dry-run mode?

Simply set the QA_FAILURES_REPORTER_OPTIONS project variable (in gitlab-org/gitlab-qa-mirror) with --dry-run.

Edited by Rémy Coutable