Multiple jobs provide input for coverage data on pipeline in Merge Request view
Problem to solve
Currently, if a pipeline has multiple jobs that output coverage, we calculate the average and use that as the coverage to report for the entire pipeline. This is almost guaranteed to be wrong, sometimes wildly so. If a developer wants more accurate coverage number, they have to jump through a bunch of hoops. I don't know how to solve this generally.
Intended users
- Sasha (Software Developer) - who wants to know from a glance at the pipeline widget on the MR page, jobs page or pipeline page how coverage has changed without having to download a report or visit another tool
- Delaney (Development Team Lead) - Who wants the project coverage badge to be an accurate reflection of coverage for the entire project.
Further details
A use case from the gitlab project.
- In the MR the pipeline shows coverage for the source branch and how it compares to the target branch.
- This value is made up of data from two jobs.
- Further complicating things is that the initial value is just the average of the data from the jobs, but there is much more ruby code than javascript in the repository.
- Even further complicating things is that there are multiple ways coverage was calculated in the karma job, we picked one.
Proposal
Modify the pipeline view that appears on the pipeline page to indicate how many jobs make up the calculated value and their coverage values in a tooltip. We may consider showing how the coverage diff from each job has changed but that may be a stretch.
See the Designs below.
Proposal options considered
Some options we have considered:
- Leave as-is
- Use only the first (or last) coverage number generated in the pipeline
- Tell people to manage their own coverage numbers and report a single, overall number.
- Document how to report a single, overall number for some set of languages/frameworks.
- Parse from various coverage reporters that know how to merge separate reports into a final number.
How we will proceed: Leave the calculated value as is and add some context from the known jobs that make up the value. Why: None of the options considered are great ones nor meet everyone's needs and there is a distinct possibility we make things worse by changing the current method. Instead we believe that calling out when a coverage value has multiple inputs and providing the context about those inputs is the best option for users. To do this there are some changes coming in other issues:
- Custom text for coverage badges will allow for a better view of coverage numbers on the project view
- Graphing code coverage over time will allow a view of how the coverage value from each job is changing over time
Other enhancements for integration with code coverage reports will be coming in the epic Make code coverage more comprehensive that we believe will further help add context to code coverage data for users.
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
If this is successful more users will include code coverage calculations in their build jobs. We will calculate how many jobs include coverage data and think that within 90 days of shipping this and the other 2 issues mentioned in the proposal customer jobs with coverage reports will increase by 10%.
What is the type of buyer?
This is a feature widely used by individuals and will be part of GitLab Core