<!-- The first four sections: "Problem to solve", "Intended users", "User experience goal", and "Proposal", are strongly recommended, while the rest of the sections can be filled out during the problem validation or breakdown phase. However, keep in mind that providing complete and relevant information early helps our product team validate the problem and start working on a solution. --> ### Release notes When you are reviewing the failed tests in a pipeline it can be hard to tell if a failing test is a legitimate failure that needs investigation or a flaky test that will pass on the next execution. The next iteration of the [repeat failed test counter](https://about.gitlab.com/releases/2021/01/22/gitlab-13-8-released/#repeat-failed-test-counter) is adding a repeat failed test counter to the unit test report. Now you can see how often a test has failed on the default branch without needing to have a Merge Request associated with the test execution. https://docs.gitlab.com/ee/ci/unit_test_reports.html#number-of-recent-failures ### Problem to solve Developers do not have an easy way to get the history of if a test has recently been passing, failing or skipped as they research a test failure in the context of a test run in a Pipeline. Without this context it is hard to point at why the build is feeling/getting slower and what test cases may to be blame for that. Without that context it's not hard to imagine Delaney and Sasha starting to skip tests and not write tests for new code. <!-- What problem do we solve? Try to define the who/what/why of the opportunity as a user story. For example, "As a (who), I want (what), so I can (why/value)." --> ### Intended users * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead) ### User experience goal <!-- What is the single user experience workflow this problem addresses? For example, "The user should be able to use the UI/API/.gitlab-ci.yml with GitLab to <perform a specific task>" https://about.gitlab.com/handbook/engineering/ux/ux-research-training/user-story-mapping/ --> The user should be able to use the UI with GitLab to see if a test has passed or failed more often over the last N executions in the JUnit report. ### Proposal This is an MVC proposal to measure if there is interest / value in this feature. If there is we will likely re-architect this to be more scalable and functional going forward. * Display a tooltip / column on the Unit Test report for the data (see mock below but designs should be done by @jj-ramirez ). * Track page views of the Unit Suite page as a snowplow event we can display in the [group dashboard](https://app.periscopedata.com/app/gitlab/633395/Testing-Category-Metrics). #### Backend The `recent_failures` relies on [pre-loaded](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/compare_test_reports_service.rb#L21) data which was only done on the MR widget's test report endpoint. So we need to do something similar for the endpoint used on the pipelines test report page. #### Design - See below. <!-- How are we going to solve the problem? Try to include the user journey! https://about.gitlab.com/handbook/journeys/#user-journey --> ### Further details <!-- Include use cases, benefits, goals, or any other details that will help us understand the problem better. --> Before investing in time to architect a real solution for this to start on the epic for [Testing History for MRs and Pipelines](https://gitlab.com/groups/gitlab-org/-/epics/4155) or even the MVC for [Test History for Projects](https://gitlab.com/gitlab-org/gitlab/-/issues/210250) we will validate that there is value and move the feature forward with this MVC. We fully expect not to re-use code and maybe interface built as part of this issue as we learn what is valuable. ### Permissions and Security <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)? Consider adding checkboxes and expectations of users with certain levels of membership https://docs.gitlab.com/ee/user/permissions.html * [ ] Add expected impact to members with no access (0) * [ ] Add expected impact to Guest (10) members * [ ] Add expected impact to Reporter (20) members * [ ] Add expected impact to Developer (30) members * [ ] Add expected impact to Maintainer (40) members * [ ] Add expected impact to Owner (50) members --> ### Documentation We should update the [documentation](https://docs.gitlab.com/ee/ci/unit_test_reports.html#number-of-recent-failures) to reflect that this message can appear on both the MR widget and the test report summary pages. <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/workflow.html#for-a-product-change * Add all known Documentation Requirements in this section. See https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements * If this feature requires changing permissions, update the permissions document. See https://docs.gitlab.com/ee/user/permissions.html --> ### Availability & Testing @zeffmorgan @rickywiens @samdbeckham - Can you ponder on some performance considerations for this section and edit accordingly please? <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> ### What does success look like, and how can we measure that? #### Acceptance Criteria * Test execution data is tracked for all failed tests and displayed on the Pipeline -> Unit Test Report * Pipeline -> Unit Test report can still load as quickly as it does today * Page Views for Unit Test report are tracked - A [separate issue has been created for this](https://gitlab.com/gitlab-org/gitlab/-/issues/235524). ``` GIVEN a junit artifact is uploaded during an MR pipeline run GIVEN there is at least one failed test in the junit report artifact WHEN you visit the pipeline page WHEN you click on the unit test report tab THEN there exists a tooltip beside the failing test indicating how many times its' failed recently ``` #### Measures of Success * This feature will be successful if we collect any feedback from internal and external users within 45 days of launch to inform the next iteration. <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> ### What is the type of buyer? The buyer for this is the developer who wants to see how a test is behaving previously to inform their current MR. <!-- What is the buyer persona for this feature? See https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/buyer-persona/ In which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> ### Is this a cross-stage feature? No. <!-- Communicate if this change will affect multiple Stage Groups or product areas. We recommend always start with the assumption that a feature request will have an impact into another Group. Loop in the most relevant PM and Product Designer from that Group to provide strategic support to help align the Group's broader plan and vision, as well as to avoid UX and technical debt. https://about.gitlab.com/handbook/product/#cross-stage-features --> ### Links / references Feedback for this MVC will be collected in the [feedback issue](https://gitlab.com/gitlab-org/gitlab/-/issues/242233). <!-- Label reminders - you should have one of each of the following labels if you can figure out the correct ones -->