Code Coverage Merge Request Approval rule
Note about availability
Please note that this new MR rule will not be available until #331001 (closed) is complete.
If you and your team want to enforce that the code coverage of your project can not decrease from a change you have had to come up with create solutions to check how the coverage changes and fail a pipeline if it decreases. While this was possible within GitLab for teams already used to Required Approvals this creates multiple workflows within the same Merge Request.
Now you can create an approval rule to enforce that code coverage does not decrease as part of a merge request so this workflow follows what the team already does.
Problem to solve
As a team lead , I want MRs that do not keep test coverage at/above the coverage of the target branch to require approval, so we can enforce as a team TDD/Quality/Coverage expectations in the project.
Users can fail the pipeline today by checking coverage in a separate job and failing the pipeline as shown in this Unfiltered recording.
- Delaney (Development Team Lead) - Wants coverage to be increasing especially in projects with low coverage.
- Sasha (Software Developer) - doesn't want to have to add bogus tests just to appease the quality gate especially when working on a critical bug fix.
User experience goal
The user should be able to see that their MR requires approval because the code coverage would decrease the project coverage.
Create a custom merge request approval rule similar to the security
Vulnerability Check approval rule. This would allow the user to opt their project into the requirement and also allow for an approval to override the block.
- If a project is setup with the MR approval rule when the MR would decrease coverage:
- Show the
Mergebutton disabled, following the same logic as when required approvals are missing. Users should not be able to interact with the button.
- Once the user fixes the code coverage issues in the MR OR there is approval to override, the MR can be merged.
- Show the
Here a POC of how a potential implementation will look like (need to be updated as it's currently relying on an existing coverage report artifact).
We will add a new merge request approval rule for code coverage and compute the percentage coverage at the pipeline level as described below.
No support for jobs with coverage at the project level.
For this first iteration of this feature we will only rely on jobs using the coverage keyword from a gitlab-ci.yml configuration file.
This will allow us to determine easily without ~performance concern which builds we need to fetch the coverage for without dealing with unfinished builds.
We would need to parse and update the coverage data in our new service for both
- Fetch builds with a coverage_regex - populated on build creation.
- Update the coverage data for those builds - builds#update_coverage
- Determine pipeline coverage percentage - pipeline#coverage
graph LR subgraph "UpdateApprovalRulesWorker" Build1[FetchBuildsWithCoverageRegex] --> Build2[BuildsUpdateCoverage] Build2[PipelineCoverage] --> Build3[GetCoveragePercentage] end
From the original author:
This is a really important thing for the coala community (coala-analyzer.org) as well as at least one commercial client of mine who's thinking about switching to gitlab.
Permissions and Security
- Add documentation / example for how to setup a coverage Merge Request Rule
Availability & Testing
What does success look like, and how can we measure that?
- Documentation for usage is written
- Demonstrate turning on/off the approval rule
- Demonstrate a pipeline passing with coverage that has 0% change or increased coverage.
- Demonstrate a pipeline failing and blocking the merge because coverage decreases
- Demonstrate a pipeline failing with a coverage decrease that is overridden
What is the type of buyer?
The Team lead is most interested in this feature to enforce team/company norms. As this leverages Merge Request Rules this will be available in GitLab Premium and above.
Is this a cross-stage feature?