Skip to content

Code Coverage Merge Request Approval rule

Note about availability

Please note that this new MR rule will not be available until #331001 (closed) is complete.

Release notes

If you and your team want to enforce that the code coverage of your project can not decrease from a change you have had to come up with create solutions to check how the coverage changes and fail a pipeline if it decreases. While this was possible within GitLab for teams already used to Required Approvals this creates multiple workflows within the same Merge Request.

Now you can create an approval rule to enforce that code coverage does not decrease as part of a merge request so this workflow follows what the team already does.

Problem to solve

As a team lead , I want MRs that do not keep test coverage at/above the coverage of the target branch to require approval, so we can enforce as a team TDD/Quality/Coverage expectations in the project.

Users can fail the pipeline today by checking coverage in a separate job and failing the pipeline as shown in this Unfiltered recording.

Intended users

User experience goal

The user should be able to see that their MR requires approval because the code coverage would decrease the project coverage.

Proposal

Create a custom merge request approval rule similar to the security Vulnerability Check approval rule. This would allow the user to opt their project into the requirement and also allow for an approval to override the block.

  • If a project is setup with the MR approval rule when the MR would decrease coverage:
    • Show the Merge button disabled, following the same logic as when required approvals are missing. Users should not be able to interact with the button.
    • Once the user fixes the code coverage issues in the MR OR there is approval to override, the MR can be merged.

backend

Here a POC of how a potential implementation will look like (need to be updated as it's currently relying on an existing coverage report artifact).

We will add a new merge request approval rule for code coverage and compute the percentage coverage at the pipeline level as described below.

First iteration

No support for jobs with coverage at the project level.

For this first iteration of this feature we will only rely on jobs using the coverage keyword from a gitlab-ci.yml configuration file.

This will allow us to determine easily without ~performance concern which builds we need to fetch the coverage for without dealing with unfinished builds.

We would need to parse and update the coverage data in our new service for both base_report and head_report:

  1. Fetch builds with a coverage_regex - populated on build creation.
  2. Update the coverage data for those builds - builds#update_coverage
  3. Determine pipeline coverage percentage - pipeline#coverage 
graph LR
  subgraph "UpdateApprovalRulesWorker"
  Build1[FetchBuildsWithCoverageRegex] --> Build2[BuildsUpdateCoverage]
  Build2[PipelineCoverage] --> Build3[GetCoveragePercentage]
end

Further details

From the original author:

This is a really important thing for the coala community (coala-analyzer.org) as well as at least one commercial client of mine who's thinking about switching to gitlab.

Permissions and Security

N/A

Documentation

  • Add documentation / example for how to setup a coverage Merge Request Rule

Availability & Testing

What does success look like, and how can we measure that?

Acceptance Criteria

  • Documentation for usage is written
  • Demonstrate turning on/off the approval rule
  • Demonstrate a pipeline passing with coverage that has 0% change or increased coverage.
  • Demonstrate a pipeline failing and blocking the merge because coverage decreases
  • Demonstrate a pipeline failing with a coverage decrease that is overridden

What is the type of buyer?

The Team lead is most interested in this feature to enforce team/company norms. As this leverages Merge Request Rules this will be available in GitLab Premium and above.

Is this a cross-stage feature?

No.

Links / references

Edited by James Heimbuck