Inline code coverage remarks inside MR diffs
Problem to solve
When a Merge Request is being reviewed comments like "I can't stamp this until you improve code coverage" can be demoralizing. Today it is impossible to tell in the GitLab UI where code is not covered by tests. This leads to a scattershot approach adding tests that may not actually improve the quality of the overall project.
Intended users
- Sasha (Software Developer) who will be creating and reviewing merge requests and has been asked to hold/improve the code coverage with each change.
Further details
To help developers see what parts of the code can be immediately improved it would be nice to have a visual representation in the diff showing where test coverage is lacking.
This would be really nice but hard to do cross-programming languages.
- Phabricator does it for PHP https://secure.phabricator.com/book/phabricator/article/arcanist_coverage/
- Atlassian does it for Java https://www.atlassian.com/software/clover/overview
- Code Climate does it in GitHub https://github.com/marketplace/code-climate
This could be shown in the Changes / Diff view this way along with Code Quality data.
Proposal
Use Cobertura to create a code coverage report on a project. On a pipeline run create utilize this visualize changes to code coverage as shown in the Designs tab.
This can be built on top of #23694
Additional community contributions are welcome to build support for additional coverage formats.
Code Quality data as shown in the original mockup will be incorporated in a follow-up issue gitlab#2526.
Permissions and Security
Documentation
- Document how to add the new job to the gitlab-ci.yml file
- Document how the code coverage information will be displayed
- Document how others can contribute additional coverage support for this feature similar to how the Package team has done this for registry contributions.
Availability & Testing
- Test for a very large code coverage file to measure performance
What does success look like, and how can we measure that?
This issue will be successful if >= 25% of Projects have included this job in their pipeline within 90 days of launch.
Acceptance Criteria
- A pipeline that includes the new job will at the same speed as before the job was added
- A merge request that includes changes to a source file includes the code coverage visualization in the diff view
What is the type of buyer?
This will be available in the Core product as the primary user is an IC.
Links / references
/label feature