Graph code coverage changes over time for a project
Problem to solve
While GitLab offers the ability to show the current Code Coverage value on a badge this does not provide the context of how that value is changing over time that several users need to do their jobs.
Persona: Delaney (Development Team Lead)
- I cannot tell how the code coverage value(s) of my team's projects is changing over time and need to for my job.
Persona: QA Manager/tester
- I cannot trust that code coverage means anything without context of historical changes so I test manually even after unit tests are run for any change going to production. This leads to me batching testing to save time but it can hold up things from deploying and people get mad at me. This sucks but is necessary for me to feel the feature won't break when real users get it.
Personal: Parker (Product Manager)
- I do not trust that an increase in code coverage means a less buggy product and I do not have data to prove that as code coverage changes customer found defects change in lock step.
Intended users
- Delaney (Development Team Lead)
- Parker (Product Manager)
- QA Manager/team member
Further details
Tracking how code coverage is changing on the master branch within a project over time allows us to move our vision for unit testing forward. By showing a graph of the change in code coverage on master in a project over time users can visualize how the overall quality is changing and start to provide context to compare to other measures like customer found defects, performance and business KPIs.
Proposal
- Add a code coverage graph over time in the Analytics -> [Repository page])(https://gitlab.com/gitlab-org/gitlab/-/graphs/master/charts). Use the value from the job(s) that include a code coverage regex.
- If there is more than one job include those job names in the dropdown so users can see coverage for each calculation.
- (For example the gitlab project has a job to calculate ruby coverage and another to calculate javascript coverage).
- For the MVC graph code coverage as defined by the user and using the name defined by the user.
- The graph will be a fixed 3 month window.
- For purposes of the MVC if a job name is changed historic data maybe lost (stretch to map from one to the other but not part of the acceptance criteria).
Outcomes
Persona: Delaney
- When I am asked about how the code coverage of a project has changed in this release, I want to have a visual representation to share, so that I can be trusted as an individual who runs a team that cares about quality.
- When I am asked about our velocity/tech debt, I want to have a visual representation to compare to bugs fixed/features shipped, so that I can be trusted as a leader running a highly performant team.
Persona: Parker
- When I am weighing bug fixes against features, I want a data point of code coverage over time of software my team works on, so that I can back up my choices about balancing features vs. fixes.
Persona: QA Manager
- When I am planning tests for a project, I want a data point of code coverage over time for the project, so I can feel confident that my test plan will find issues before customers do.
Permissions and Security
Documentation
- Update code coverage to instruct how to make the graph(s) appear if need be
- Document how the names used in code coverage definition will appear on the graph to set expectations with customers.
- Update code coverage documentation to reflect where the graph appears and what data will appear when (eg; it will update daily for yesterday's last code coverage value for /master).
- Note that if a job name is changed historic tracking will be lost per discussion in the issue.
- Document that data points are available and shown only on days the job to calculate the coverage is completed.
Testing
- Check load time on page to ensure this is performant / doesn't slow the page load time down.
- Check data quality / graph look.
What does success look like, and how can we measure that?
We want both Delaney and our QA engineer to be confident that the trajectory of the code coverage is positive with changes being made while balancing the desire of a product manager to deliver value to users through new features. This graph should deliver confidence to all 3 parties by giving Delaney a measure for engineers to write tests with new code, QA to have confidence that as changes are introduced basic unit tests are also being added and the PM to know that new features will be available / work as intended for users.
- Page views of graph page. We believe this feature will result in an increase of page views on the Repository Analytics page by 10% on instances that can utilize the feature.
Acceptance Criteria
- Graph is displayed for projects with code coverage stats.
- Show last data point for the day and scale of days?
- Data displayed is accurate as compared to raw data
- Cross browser testing / things look acceptable to design (Making it look nice on mobile is a stretch for the MVC)
What is the type of buyer?
The buyer for this feature is likely Delaney but we see this iteration as basic functionality that belongs in our Core offering. As more features are developed on top of this MVC they may be part of a different tier.
Links / references
This is an iteration on the original Graph testing results over time MVC issue #1020 (closed).![Project_Details]