Update test reporting to dogfood Test Cases feature and reduce manual SET intervention
This uses the new Test Cases feature to address the memory issues that occur when testcase issues have too many notes and also removes the need for SETs to manually create new testcase issues and open MRs to update testcase issue links in the code. (I capitalized Test Cases here to differentiate the new Test Cases feature from the current testcase issues.)
In this proposal, we would use Test Cases as "hubs" and the current testcase issues as "reporting" issues. For each test, the hardcoded testcase link in the code would now point to an actual Test Case which would link to a separate reporting issue that is updated with status labels and stacktraces. If the script doesn't find a reporting issue that matches by file path & title, it will create a new one and link it to this Test Case.
(Part 5) When the reporting issue reaches a certain note threshold, it will automatically be closed and a new one opened with the same metadata and will also be linked to the Test Case. (This could be a separate script that runs say weekly?) This way when new reporting issues are needed, we don't have to manually create a new issue and open an MR to update the testcase issue link in the code. This will also close out the reporting issue before it gets too big to open, thus preserving the test run history and preventing memory issues.
Tasks:
-
Part 1: Ensure there is a Test Case and issue with correct title created for each test - https://gitlab.com/gitlab-org/quality/testcases/-/quality/test_cases?label_name[]=status%3A%3Aautomated -
Part 2: Temporarily disable the reporting script !746 (merged) (Test sessions will still be available for pipeline triage purposes) -
Part 3: Update the hardcoded testcase issue links in the code to point to the actual Test Cases - gitlab!68677 (merged) Once this has reached production, the next can be merged: -
Part 4: Update the reporting script to parse the testcase link out of the Test Case description and to automatically add a link in the description(Test Cases don't have the linked issues widget) of the Test Case to the reporting issue it finds/creates if not linked already. - !735 (merged) -
Part 5: Add check to reporting issues for note count(x-total as it's quick to retrieve without getting all the notes) and create new ones as necessary. gitlab-org/quality/toolbox!86 (merged)
Notes: After step 4 is complete, when a result issue has too many notes and times out, it just needs to be closed. A new one will be automatically created and linked to the proper Test Case. Example Test Case. Once step 5 is complete, this process will be automated.
Original discussion: