Test Coverage Visualization doesn't easily support C# and Go cobertura reports
Problem to solve
As a software engineer, when I configure the inline code coverage feature and upload a cobertura report generated by a common library in C# and Go it should "just work" so that I can get back to coding and not worry about parsing a report with a program before I pass it to GitLab.
Right now the Test Coverage Visualization feature doesn't handle paths in the cobertura report in a robust way. Many different coverage report generating libraries have different approaches around:
- How much of the path is included in the report in general
- Whether the path is recorded as a relative or an absolute path
- Where that path is in the cobertura file (the amount of the path that is in the
sourcenode as compared to the
classnode in the
This is particularly troublesome for GitLab CI as the report is typically generated inside of a container, and then processed inside of the main application. This means that the filesystem for the report generation is completely different than the filesystem used to process the report.
The result of this is several notable report generating libraries will generate reports incompatible with this feature some of the time. It is unknown how often this is the case, but there have been a few issues raised:
- C#/coverlet: #213444 (closed)
- Python/coverage: https://forum.gitlab.com/t/enabling-cobertura/36918/5
We can build some logic into how the paths are handled before we attempt to match them with files in the merge request. Ideally we could find a way to intelligently interpret the path so we can get to the project root path and not consider the rest of the path when attempting a match.
We should look into implementing some logic around matching absolute paths and paths from the root of the git project, so that the cobertura report for C# can be parsed when it is generated in a container.
@rickywiens has an example C# project, and both https://gitlab.com/gitlab-org/gitaly and https://gitlab.com/gitlab-org/gitlab-runner are using workarounds to fix the pathing issues, so they would be good samples to use for this.
An issue has been created to document the process for working around this issue: #217365 after this is fixed that documentation should be removed/modified.
Availability & Testing
We should try and test all known combinations/permutations of the paths: Relative/Absolute, various splits between the path being in the
source node and paths being in the
filename attribute in the
class node. We should likely generate example cobertura reports from popular libraries and include them in our integration tests for the feature.
Some ideas about popular libraries:
- probably more?
What does success look like, and how can we measure that?
- Test Coverage Visualization "works" (meets original feature AC) for projects using C# or Go Libraries and generating a cobertura report.
- Documentation is updated to detail any special setup / changes that need to be made to a project or report to make the feature work.
Measures of success
- Increase in page views of the feature documentation page by 10% Month over Month after release (so if this ships on Sept 22 we would measure Oct vs. Sept) to measure any increase in awareness of the feature.
- There is no tracking of feature usage at this time.
We should make this feature to "just work" with the dominant C# and Go libraries that generate cobertura reports.