Skip to content

Test Coverage Visualization doesn't easily support C# cobertura reports

Problem to solve

As a software engineer, when I configure the inline code coverage feature and upload a cobertura report generated by a common library in C# it should "just work" so that I can get back to coding and not worry about parsing a report with a program before I pass it to GitLab.

Right now the Test Coverage Visualization feature doesn't handle paths in the cobertura report in a robust way. Many different coverage report generating libraries have different approaches around:

  1. How much of the path is included in the report in general
  2. Whether the path is recorded as a relative or an absolute path
  3. Where that path is in the cobertura file (the amount of the path that is in the source node as compared to the class node in the filename attribute varies

This is particularly troublesome for GitLab CI as the report is typically generated inside of a container, and then processed inside of the main application. This means that the filesystem for the report generation is completely different than the filesystem used to process the report.

The result of this is several notable report generating libraries will generate reports incompatible with this feature some of the time. It is unknown how often this is the case, but there have been a few issues raised:

  1. C#/coverlet: #213444 (closed)
  2. Python/coverage: https://forum.gitlab.com/t/enabling-cobertura/36918/5

Intended users

Proposal

We can build some logic into how the paths are handled before we attempt to match them with files in the merge request. Ideally we could find a way to intelligently interpret the path so we can get to the project root path and not consider the rest of the path when attempting a match.

We should look into implementing some logic around matching absolute paths and paths from the root of the git project, so that the cobertura report for C# can be parsed when it is generated in a container.

@rickywiens has an example C# project, and both https://gitlab.com/gitlab-org/gitaly and https://gitlab.com/gitlab-org/gitlab-runner are using workarounds to fix the pathing issues, so they would be good samples to use for this.

Further details

Documentation

An issue has been created to document the process for working around this issue: #217365 (closed) after this is fixed that documentation should be removed/modified.

Availability & Testing

We should try and test all known combinations/permutations of the paths: Relative/Absolute, various splits between the path being in the source node and paths being in the filename attribute in the class node. We should likely generate example cobertura reports from popular libraries and include them in our integration tests for the feature.

Some ideas about popular libraries:

  1. ruby/rspec
  2. python/coverage
  3. Java/??
  4. C#/coverlet
  5. probably more?

What does success look like, and how can we measure that?

Acceptance Criteria

  • Test Coverage Visualization "works" (meets original feature AC) for projects using C# Libraries and generating a cobertura report.
  • Documentation is updated to detail any special setup / changes that need to be made to a project or report to make the feature work.

Measures of success

  • Increase in page views of the feature documentation page by 10% Month over Month after release (so if this ships on Sept 22 we would measure Oct vs. Sept) to measure any increase in awareness of the feature.
  • There is no tracking of feature usage at this time.

We should make this feature to "just work" with the dominant C# libraries that generate cobertura reports.

What is the type of buyer?

Is this a cross-stage feature?

Links / references

This will be behind a feature flag: :smart_cobertura_parser

Edited by Ricky Wiens