Deduplicate QA failure issues and link to testcase
We have a tool that creates/updates testcase issues to report the status of test results. (See https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/187).
However, it's still a laborious process to monitor the QA pipelines and create/update issues to identify and resolve failures.
Initial iteration
- Detect known QA failure issues when a test fails and create a new issue otherwise. => gitlab-org/gitlab-qa!546 (merged)
- Link the (existing or new) failure issue to the testcase issue. => gitlab-org/gitlab-qa!546 (merged)
- (Stretch goal): Automatically assign the pipeline triage DRIs and cc the secondaries
- Add a comment with the failing job URL to the QA failure issue to gather occurrences of failures (not sure about that as this could lead to a flood of notes). => gitlab-org/gitlab-qa!546 (merged)
- Update
gitlab-org/gitlab-qa
's.gitlab-ci.yml
(used formaster
pipelines as well as post-deployment QA pipelines). => gitlab-org/gitlab-qa!546 (merged)- Set
QA_FAILURES_REPORTER_OPTIONS: "--dry-run"
for now.
- Set
- Release a new
gitlab-qa
version and remove the--dry-run
option: gitlab-org/gitlab-qa!565 (merged) - Update
gitlab-org/quality/nightly
's.gitlab-ci.yml
(https://gitlab.com/gitlab-org/quality/nightly/-/blob/4e6215f9cd7319f1988e970921fe7817a599bd1e/.gitlab-ci.yml#L94) => https://gitlab.com/gitlab-org/quality/nightly/-/merge_requests/73 - Update
gitlab-org/quality/pipeline-common
's/ci/qa.yml
(https://ops.gitlab.net/gitlab-org/quality/pipeline-common/-/blob/c25125ed38f6d7b0e644f5d77b690df8e3b52a97/ci/qa.yml#L100) => gitlab-org/quality/pipeline-common!66 (merged)
Follow-up iterations
-
Put link to the QA failure issues in the QA session report issue (next to the testcase issue link in the tables). Done!
Former proposals:
- Post a Slack message with a link to the failure issue (as a reply to the pipeline failure message that currently gets posted automatically).
- Add a comment in the pipeline triage report issue.
Challenges
- Duplicate detection. If an open failure issue already exists we shouldn't open a duplicate. And if a closed issue exists but is still relevant, in some circumstances we'll want to reopen it.
- We should be able to overcome this challenge using the duplicate-handling in the testcase issue reporter - it's capable of updating the existing issue relevant to a particular test. Although it does have problems with stacktraces that aren't identical even if they're ultimately the same failure.
Edited by Rémy Coutable