Project level Test Result history for default branch
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem to solve
Team leads and SETs cannot get at a glance data about how many tests have passed in the last x builds or over the last x days for their project. This limits their ability to strike the right balance between building new features and fixing bugs.
Intended users
- Parker (Product Manager) - who is pushing for new features in the absence of data about failing tests
- Delaney (Development Team Lead) - who knows that the team is saying tests are failing more often.
- Simone (Software Engineer in Test) - who knows that regression tests are taking longer and failing more often but has to write down findings to prove it.
Further details
Customers migrating from other CI systems expect a view of the last x builds and tests within them showing pass/fail data. This data can later be used to detect flaky tests.
Proposal
- Add a view to Analytics -> CI/CD Charts that shows some stats for tests from the last x builds such as:
- For each of the last 10 pipeline runs for the default branch of a project store the list of tests that failed. stretch: store skipped tests too
- If a test isn't noted for a pipeline we can assume it passed (but what if it didn't exist in that pipeline?) but we'll need to know about all tests somehow, probably by parsing the latest junit report?
- stretch: store the time it took to run the test too.
- Show a table of the tests that have failed most often (sort by # of failures descending) with a history of the last 10 pipeline executions for that test.
- Let a user click through to each pipeline execution, the target is the test tab and ideally the job that test is in. If #217514 (closed) which filters to failed tests by default is done this view will be a lot better but that's not a blocker to getting this view done.
- Performance - If loading ALL the tests on a single page or even to show in a paginated view is not performant let's consider with @jj-ramirez showing jobs first then digging into tests by job.
Design is being iterated on in #223737 (closed) and final designs will be added to this issue later.
Permissions and Security
No new permissions needed.
Documentation
- Document where the data appears on the JUnit test reports page
Availability & Testing
What does success look like, and how can we measure that?
Acceptance Criteria
- The analytics page should load in only a second or two slower than it did previously, even for projects with large test suites like GitLab
- A user should be able to navigate to a failed test for any of their last 10 pipelines from the page
- Any user at the Starter or above should be able to see the report.
What is the type of buyer?
Marking as GitLab Ultimate as this ties into the Value stream analytics capabilities
Links / references
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.