Generate test session report, summary of testcase info, pipeline info, linked release QA issue and triage checkbox

There is a need to have a birds eye view of all the end-to-end tests run for a given release.

Proposal

Automated a pairing issue that can be linked to the corresponding QA release issue. But instead of a list of MRs, list the list of testcases from https://gitlab.com/gitlab-org/quality/testcases/ that was ran for that deployment on staging.

See example in gitlab-org/quality/testcase-sessions#1

A issue that contains a list of testcases linked https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/504
For each testcase link to the pipeline location where that testcase ran
Apply pass fail label at the end.
For each testcase entry with fail provide a checkbox for quick triage if that test re-run passed.
Generate report for each deployment to staging.
Link to related release QA issue

This can be linked from https://gitlab.com/gitlab-org/release/tasks/-/issues/1491#automated-qa

Brainstorming

Also attached is the brainstorming session I had with @jo_shih https://drive.google.com/drive/folders/1j1T8DFFvuC0m7txbCFhnIXjBOpqeHB6H

Works being done:

Changes needed for general information:

Code cleanup to make it more extensible: gitlab-org/gitlab-qa!533 (merged)
Persist testcase into artifacts so we have them available while generating the report: gitlab-org/gitlab-qa!534 (merged)
Pass the needed CI_JOB_URL we need to show in the report, also cleanup more codes: gitlab-org/gitlab-qa!543 (merged) (This requires gitlab-org/gitlab!44762 (merged) as well)

Changes needed for retrieving deployment information and QA issue

Provide the actual QA issue URL after we generated the QA issue: gitlab-org/release-tools!1232 (merged)
Provide a way to retrieve the job id after we trigger a specific pipeline while deployment: https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/315 (merged)
The integral changes to collect the QA issue URL, and pass the deployment information along way down to the QA pipeline for showing the deployment name and connect the QA issue: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/319 (reverted)

The last integral part for putting everything together

gitlab-org/gitlab-qa!526 (merged) made several features:
- The latest generation: gitlab-org/quality/testcase-sessions#16 (closed)
- Short summary for all the stages
- Only list failed tests, collapsing all other tests by default
- Group tests with the same testcase
- Connect the related failure issues for the test (based on gitlab-org/gitlab-qa!546 (merged))
- Connect the QA issue (waiting for https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/319 (reverted))
- Update the Slack notification to contain the URL to the generated test report

Follow ups

Edited Nov 06, 2020 by Lin Jen-Shin