Test History MVC - Test Summary Widget
Problem to solve
Developers do not have an easy way to get the history of if a test has recently been passing, failing or skipped as they research a test failure in the context of a test run in an MR.
Without this context it is hard to point at why the build is feeling/getting slower and what test cases may to be blame for that. Without that context it's not hard to imagine Delaney and Sasha starting to skip tests and not write tests for new code.
Intended users
User experience goal
The user should be able to use the UI with GitLab to see if a test has passed or failed more often over the last N executions in the default branch in the test summary.
Proposal
This is an MVC proposal to measure if there is interest / value in this feature. If there is we will likely re-architect this to be more scalable and functional going forward.
- Count failures by test for any test execution in the project on the default
branch
- We changed course to use the
default
branch for comparison and realize it may limit what we can learn in the MVC but the implementation was made easier so we can learn faster. - If a job/pipeline generates a lot of errors (TBD how many by Back end developer) Stop counting and do not increment the count for existing tests. This is likely the result of something else that went wrong and this would lead to noisy data for users. For Example the database was down so all the tests are failing, we shouldn't count that as a test failure.
- We changed course to use the
- This data will be sent along with the test case entity already used by the Test Summary Widget.
- Display how many times a test has failed over the last X days (history of the test) for each failed test that has a previous failure recorded in the /default branch.
- Track expansions of the Test Summary widget as a snowplow event we can display in the group dashboard.
Application Logic
Backend
After parsing a JUnit XML report file, if:
- The feature flag is enabled for this project
- The file contains less than 200 total failures
We will increment a persisted counter for each test that failed.
We will create an endpoint/method that will retrieve a count of how many times a test has failed in the last 14 days.
Frontend
Upon loading the MR page, if:
- The feature flag is enabled for this project
- The JUnit XML report file is done processing
- The JUnit XML report file contains failures
- The JUnit XML report failures are loaded into the MR widget
We will build the recent failures data into the data for the MR Widget, then draw the tooltips on the frontend with that information.
Postgres
We will store the failures and test cases in the database in tables like:
ci_test_cases(project_id, test_case_name)
has_many
ci_test_case_failures(ci_test_case_id, failed_at, ref)
Then query those tables to create the necessary endpoints as described here: #241759 (comment 417521675)
- For the
test_case_name
we'll need a reasonable limit (text limit). - I suppose
failed_at
is needed for doing statistical calculations, since the same data is probably available in one of the CI tables. - For inserting records in batches, check out: https://docs.gitlab.com/ee/development/insert_into_tables_in_batches.html
- It might make sense at some point to aggregate and store the failure counts (last 14 days) on the
ci_test_cases
table.
Other possible technical implementations are discussed here: #235525 (comment 394298215)
Further details
Before investing in time to architect a real solution for this to start on the epic for Testing History for MRs and Pipelines or even the MVC for Test History for Projects we will validate that there is value and move the feature forward with this MVC. We fully expect not to re-use code and maybe interface built as part of this issue as we learn what is valuable.
The backend should be complete in this issue, which should then be re-usable in #235525 (closed) which only has a frontend label.
Permissions and Security
Documentation
- Add a note to existing documentation that number of times a test has failed previously is now inclued.
- Update the screenshot for the newly refined UI.
Availability & Testing
What does success look like, and how can we measure that?
Acceptance Criteria
- Test execution data is tracked for all failed tests and displayed on the test summary MR Widget
- MR can still load as quickly as it does today
- Opens of the Test Summary Widget are tracked - A [separate issue has been created for this]
GIVEN a junit artifact is uploaded during an MR pipeline run
GIVEN there is at least one failed test in the junit report artifact
WHEN you visit the MR page
WHEN you expand the test summary in the MR widget section
THEN there exists a tooltip beside the failing test indicating how many times its' failed recently
Measures of Success
- This feature will be successful if there are at least 500 opens of the test widget per week the first full week 30 days after launch.
- This feature will be successful if we collect any feedback from internal and external users within 45 days of launch to inform the next iteration.
What is the type of buyer?
The buyer for this is the developer who wants to see how a test is behaving previously to inform their current MR.
Is this a cross-stage feature?
No.
Links / references
Feedback for this MVC will be collected in the feedback issue.
Follow-up Issues to create:
-
Create daily clean-up job to ensure failure table has 14 days of data only -
Add per-pipeline limit of 1000 failures stored -
Add per-project limit of 100k failures stored at once -
Aggregate failures in last 14 days in background job and persist into ci_test_case table to avoid expensive GROUP BY COUNT queries