Generate test session report, summary of testcase info, pipeline info, linked release QA issue and triage checkbox
There is a need to have a birds eye view of all the end-to-end tests run for a given release.
Proposal
Automated a pairing issue that can be linked to the corresponding QA release issue. But instead of a list of MRs, list the list of testcases from https://gitlab.com/gitlab-org/quality/testcases/ that was ran for that deployment on staging.
See example in gitlab-org/quality/testcase-sessions#1
- A issue that contains a list of testcases linked https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/504
- For each testcase link to the pipeline location where that testcase ran
- Apply
pass
fail
label at the end. - For each testcase entry with
fail
provide a checkbox for quick triage if that test re-run passed. - Generate report for each deployment to staging.
- Link to related release QA issue
This can be linked from https://gitlab.com/gitlab-org/release/tasks/-/issues/1491#automated-qa
Brainstorming
Also attached is the brainstorming session I had with @jo_shih https://drive.google.com/drive/folders/1j1T8DFFvuC0m7txbCFhnIXjBOpqeHB6H
Works being done:
Changes needed for general information:
- Code cleanup to make it more extensible: gitlab-org/gitlab-qa!533 (merged)
- Persist testcase into artifacts so we have them available while generating the report: gitlab-org/gitlab-qa!534 (merged)
- Pass the needed
CI_JOB_URL
we need to show in the report, also cleanup more codes: gitlab-org/gitlab-qa!543 (merged) (This requires gitlab-org/gitlab!44762 (merged) as well)
Changes needed for retrieving deployment information and QA issue
- Provide the actual QA issue URL after we generated the QA issue: gitlab-org/release-tools!1232 (merged)
- Provide a way to retrieve the job id after we trigger a specific pipeline while deployment: https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/315 (merged)
- The integral changes to collect the QA issue URL, and pass the deployment information along way down to the QA pipeline for showing the deployment name and connect the QA issue: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/319 (reverted)
The last integral part for putting everything together
-
gitlab-org/gitlab-qa!526 (merged) made several features:
- The latest generation: gitlab-org/quality/testcase-sessions#16 (closed)
- Short summary for all the stages
- Only list failed tests, collapsing all other tests by default
- Group tests with the same testcase
- Connect the related failure issues for the test (based on gitlab-org/gitlab-qa!546 (merged))
- Connect the QA issue (waiting for https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/319 (reverted))
- Update the Slack notification to contain the URL to the generated test report
Follow ups
Edited by Lin Jen-Shin