Use GitLab issues for test case management and reporting with data from CI
We need to provide more visibility to our test coverage and the results of these tests. In support of our focus on dogfooding and building functionality into GitLab, we could use an additional CI job that iterates on the jUnit test result xml files generated. Using that information, update, create, and label issues that represent testcases and their runs through our existing API calls.
Discussion: https://www.youtube.com/watch?v=vtaIblbWplA
We now have https://gitlab.com/gitlab-org/quality/testcases/ where:
- Each issue represents a testcase
- E.g. this issue represents a UI automated test for
Creating a Merge Request
gitlab-org/quality/testcases#1 (closed)
- E.g. this issue represents a UI automated test for
- Test type labels are used to define the type of the test.
- ~api-test: A rest API test
- ~ui-test: A Selenium UI test
- ~visual-test: A visual test
- Test result status labels are used to define current execution result of the test.
- ~"staging:failing": test is failing on staging
- ~"staging:passing": test is passing on staging
- ~"master:failing": test is failing on latest
master
branch - ~"master:passing": test is passing on latest
master
branch
- We can use boards to visualize the results of the test for now. https://gitlab.com/gitlab-org/quality/testcases/boards/988752
- We can use issue APIs or direct controller access on a new CI job type to update the result of a test and set result status labels.
Proposal
- [ ] Allow the test report JSON to be retrieved via the API - gitlab-org/gitlab!24648 (merged) (no longer required but would still be useful in a later iteration to make the reporter more efficient)
-
Use CI job to iterate test report JSON to generate list of testcases executed and their results - https://gitlab.com/gitlab-org/quality/team-tasks/issues/387 -
Updated the related issue in the matching Project through our API - match an RSpec tag to project_id/issue_number? -
Automatically label issues with context::result labels to have them displayed in our issue boards
There are many possible other ideas to flesh out here, but this would likely represent what we would be looking for in an MVP.
/cc @gl-quality