Skip to content
Snippets Groups Projects
Commit 6ae895e4 authored by Joe Randazzo's avatar Joe Randazzo :anger_right: Committed by Catherine Pope
Browse files

Improve organization of code coverage documentation

parent 158b2a5a
No related branches found
No related tags found
2 merge requests!118700Remove refactor_vulnerability_filters feature flag,!117058Improve organization of code coverage documentation
Showing
with 162 additions and 143 deletions
......@@ -208,119 +208,7 @@ Jobs that exceed the timeout are marked as failed.
You can override this value [for individual runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner).
## Merge request test coverage results
If you use test coverage in your code, you can use a regular expression to
find coverage results in the job log. You can then include these results
in the merge request in GitLab.
If the pipeline succeeds, the coverage is shown in the merge request widget and
in the jobs table. If multiple jobs in the pipeline have coverage reports, they are
averaged.
![MR widget coverage](img/pipelines_test_coverage_mr_widget.png)
![Build status coverage](img/pipelines_test_coverage_build.png)
### Add test coverage results using `coverage` keyword
To add test coverage results to a merge request using the project's `.gitlab-ci.yml` file, provide a regular expression
using the [`coverage`](../yaml/index.md#coverage) keyword.
### Test coverage examples
Use this regex for commonly used test tools.
<!-- vale gitlab.Spelling = NO -->
- Simplecov (Ruby). Example: `/\(\d+.\d+\%\) covered/`.
- pytest-cov (Python). Example: `/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/`.
- Scoverage (Scala). Example: `/Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)/`.
- `pest --coverage --colors=never` (PHP). Example: `/^\s*Cov:\s*\d+\.\d+?%$/`.
- `phpunit --coverage-text --colors=never` (PHP). Example: `/^\s*Lines:\s*\d+.\d+\%/`.
- gcovr (C/C++). Example: `/^TOTAL.*\s+(\d+\%)$/`.
- `tap --coverage-report=text-summary` (NodeJS). Example: `/^Statements\s*:\s*([^%]+)/`.
- `nyc npm test` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`.
- `jest --ci --coverage` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`.
- excoveralls (Elixir). Example: `/\[TOTAL\]\s+(\d+\.\d+)%/`.
- `mix test --cover` (Elixir). Example: `/\d+.\d+\%\s+\|\s+Total/`.
- JaCoCo (Java/Kotlin). Example: `/Total.*?([0-9]{1,3})%/`.
- `go test -cover` (Go). Example: `/coverage: \d+.\d+% of statements/`.
- .NET (OpenCover). Example: `/(Visited Points).*\((.*)\)/`.
- .NET (`dotnet test` line coverage). Example: `/Total\s*\|\s*(\d+(?:\.\d+)?)/`.
- tarpaulin (Rust). Example: `/^\d+.\d+% coverage/`.
- Pester (PowerShell). Example: `/Covered (\d+\.\d+%)/`.
<!-- vale gitlab.Spelling = YES -->
### View code coverage history
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10.
> - Graph [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1.
To see the evolution of your project code coverage over time,
you can view a graph or download a CSV file with this data.
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Analytics > Repository**.
The historic data for each job is listed in the dropdown list above the graph.
To view a CSV file of the data, select **Download raw data (`.csv`)**.
![Code coverage graph of a project over time](img/code_coverage_graph_v13_1.png)
Code coverage data is also [available at the group level](../../user/group/repositories_analytics/index.md).
### Coverage check approval rule **(PREMIUM)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15765) in GitLab 14.0.
> - [Made configurable in Project Settings](https://gitlab.com/gitlab-org/gitlab/-/issues/331001) in GitLab 14.1.
You can implement merge request approvals to require approval by selected users or a group
when merging a merge request would cause the project's test coverage to decline.
Follow these steps to enable the `Coverage-Check` MR approval rule:
1. Set up a [`coverage`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value.
1. Go to your project and select **Settings > Merge requests**.
1. Under **Merge request approvals**, select **Enable** next to the `Coverage-Check` approval rule.
1. Select the **Target branch**.
1. Set the number of **Approvals required** to greater than zero.
1. Select the users or groups to provide approval.
1. Select **Add approval rule**.
![Coverage-Check approval rule](img/coverage_check_approval_rule_14_1.png)
### Remove color codes from code coverage
Some test coverage tools output with ANSI color codes that aren't
parsed correctly by the regular expression. This causes coverage
parsing to fail.
Some coverage tools don't provide an option to disable color
codes in the output. If so, pipe the output of the coverage tool through a
small one line script that strips the color codes off.
For example:
```shell
lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g'
```
## Pipeline badges
You can use [pipeline badges](../../user/project/badges.md) to indicate the pipeline status and
test coverage of your projects. These badges are determined by the latest successful pipeline.
<!-- ## Troubleshooting
Include any troubleshooting steps that you can foresee. If you know beforehand what issues
one might have when setting this up, or when something is changed, or on upgrading, it's
important to describe those, too. Think of things that may go wrong and include them here.
This is important to minimize requests for support, and to avoid doc comments with
questions that you know someone might ask.
Each scenario can be a third-level heading, for example `### Getting error message X`.
If you have none to add when creating a doc, leave this section in place
but commented out to help encourage others to add to it in the future. -->
---
stage: Verify
group: Pipeline Execution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Code coverage **(FREE)**
Use code coverage to provide insights on what source code is being validated by a test suite. Code coverage is one of many test metrics that can determine software performance and quality.
## View Code Coverage results
Code Coverage results are shown in:
- Merge request widget
- Project repository analytics
- Group repository analytics
- Repository badge
For more information on test coverage visualization in the file diff of the MR, see [Test Coverage Visualization](test_coverage_visualization.md).
### View code coverage results in the MR
If you use test coverage in your code, you can use a regular expression to
find coverage results in the job log. You can then include these results
in the merge request in GitLab.
If the pipeline succeeds, the coverage is shown in the merge request widget and
in the jobs table. If multiple jobs in the pipeline have coverage reports, they are
averaged.
![MR widget coverage](img/pipelines_test_coverage_mr_widget.png)
![Build status coverage](img/pipelines_test_coverage_build.png)
#### Add test coverage results using `coverage` keyword
To add test coverage results to a merge request using the project's `.gitlab-ci.yml` file, provide a regular expression
using the [`coverage`](../yaml/index.md#coverage) keyword.
#### Test coverage examples
Use this regex for commonly used test tools.
<!-- vale gitlab.Spelling = NO -->
- Simplecov (Ruby). Example: `/\(\d+.\d+\%\) covered/`.
- pytest-cov (Python). Example: `/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/`.
- Scoverage (Scala). Example: `/Statement coverage[A-Za-z\.*]\s*:\s*([^%]+)/`.
- `pest --coverage --colors=never` (PHP). Example: `/^\s*Cov:\s*\d+\.\d+?%$/`.
- `phpunit --coverage-text --colors=never` (PHP). Example: `/^\s*Lines:\s*\d+.\d+\%/`.
- gcovr (C/C++). Example: `/^TOTAL.*\s+(\d+\%)$/`.
- `tap --coverage-report=text-summary` (NodeJS). Example: `/^Statements\s*:\s*([^%]+)/`.
- `nyc npm test` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`.
- `jest --ci --coverage` (NodeJS). Example: `/All files[^|]*\|[^|]*\s+([\d\.]+)/`.
- excoveralls (Elixir). Example: `/\[TOTAL\]\s+(\d+\.\d+)%/`.
- `mix test --cover` (Elixir). Example: `/\d+.\d+\%\s+\|\s+Total/`.
- JaCoCo (Java/Kotlin). Example: `/Total.*?([0-9]{1,3})%/`.
- `go test -cover` (Go). Example: `/coverage: \d+.\d+% of statements/`.
- .NET (OpenCover). Example: `/(Visited Points).*\((.*)\)/`.
- .NET (`dotnet test` line coverage). Example: `/Total\s*\|\s*(\d+(?:\.\d+)?)/`.
- tarpaulin (Rust). Example: `/^\d+.\d+% coverage/`.
- Pester (PowerShell). Example: `/Covered (\d+\.\d+%)/`.
<!-- vale gitlab.Spelling = YES -->
### View history of project code coverage
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10.
> - Graph [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1.
To see the evolution of your project code coverage over time,
you can view a graph or download a CSV file with this data.
1. On the top bar, select **Main menu > Projects** and find your project.
1. On the left sidebar, select **Analytics > Repository**.
The historic data for each job is listed in the dropdown list above the graph.
To view a CSV file of the data, select **Download raw data (`.csv`)**.
![Code coverage graph of a project over time](img/code_coverage_graph_v13_1.png)
### View history of group code coverage **(PREMIUM)**
To see the all the project's code coverage under a group over time, you can find view [group repository analytics](../../user/group/repositories_analytics/index.md).
![Code coverage graph of a group over time](img/code_coverage_group_report.png)
### Pipeline badges
You can use [pipeline badges](../../user/project/badges.md#test-coverage-report-badges) to indicate the pipeline status and
test coverage of your projects. These badges are determined by the latest successful pipeline.
## Coverage check approval rule **(PREMIUM)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15765) in GitLab 14.0.
> - [Made configurable in Project Settings](https://gitlab.com/gitlab-org/gitlab/-/issues/331001) in GitLab 14.1.
When merging a request that would cause the project's test coverage to decline, you can stipulate that such merge requests require approval by selected users or a group.
Follow these steps to enable the `Coverage-Check` MR approval rule:
1. Set up a [`coverage`](../yaml/index.md#coverage) regular expression for all jobs you want to include in the overall coverage value.
1. Go to your project and select **Settings > Merge requests**.
1. Under **Merge request approvals**, select **Enable** next to the `Coverage-Check` approval rule.
1. Select the **Target branch**.
1. Set the number of **Approvals required** to greater than zero.
1. Select the users or groups to provide approval.
1. Select **Add approval rule**.
![Coverage-Check approval rule](img/coverage_check_approval_rule_14_1.png)
## Troubleshooting
### Remove color codes from code coverage
Some test coverage tools output with ANSI color codes that aren't
parsed correctly by the regular expression. This causes coverage
parsing to fail.
Some coverage tools do not provide an option to disable color
codes in the output. If so, pipe the output of the coverage tool through a one-line script that strips the color codes.
For example:
```shell
lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g'
```
doc/ci/testing/img/code_coverage_group_report.png

301 KiB

......@@ -14,6 +14,7 @@ display reports or link to important information directly from [merge requests](
| [Accessibility Testing](accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. |
| [Browser Performance Testing](browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. |
| [Load Performance Testing](load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. |
| [Code Coverage](code_coverage.md) | See code coverage results in the MR, project or group. |
| [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. |
| [Display arbitrary job artifacts](../yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../jobs/job_artifacts.md) in merge requests. |
| [Unit test reports](unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. |
......
......@@ -54,8 +54,8 @@ of times the line was checked by tests.
Uploading a test coverage report does not enable:
- [Test coverage results in merge requests](../pipelines/settings.md#merge-request-test-coverage-results).
- [Code coverage history](../pipelines/settings.md#view-code-coverage-history).
- [Test coverage results in merge requests](code_coverage.md#view-code-coverage-results-in-the-mr).
- [Code coverage history](code_coverage.md#view-history-of-project-code-coverage).
You must configure these separately.
......
......@@ -1512,6 +1512,7 @@ In this example:
**Additional details**:
- You can find parse examples in [Code Coverage](../testing/code_coverage.md#test-coverage-examples).
- If there is more than one matched line in the job output, the last line is used
(the first result of reverse search).
- If there are multiple matches in a single line, the last match is searched
......
......@@ -85,10 +85,10 @@ The following are unavailable compliance violations that are tracked in [epic 52
|:-------------------------------------|:---------------|:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------|
| Pipeline failed | Medium | [Pipeline results](../../../ci/pipelines/index.md) | Merge requests pipeline failed and was merged. |
| Pipeline passed with warnings | Info | [Pipeline results](../../../ci/pipelines/index.md) | Merge request pipeline passed with warnings and was merged. |
| Code coverage down more than 10% | High | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of more than 10%. |
| Code coverage down between 5% to 10% | Medium | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of between 5% to 10%. |
| Code coverage down between 1% to 5% | Low | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of between 1% to 5%. |
| Code coverage down less than 1% | Info | [Code coverage](../../../ci/pipelines/settings.md#merge-request-test-coverage-results) | Code coverage report for the merge request indicates a reduction in coverage of less than 1%. |
| Code coverage down more than 10% | High | [Code coverage](../../../ci/testing/code_coverage.md#view-code-coverage-results-in-the-mr) | Code coverage report for the merge request indicates a reduction in coverage of more than 10%. |
| Code coverage down between 5% to 10% | Medium | [Code coverage](../../../ci/testing/code_coverage.md#view-code-coverage-results-in-the-mr) | Code coverage report for the merge request indicates a reduction in coverage of between 5% to 10%. |
| Code coverage down between 1% to 5% | Low | [Code coverage](../../../ci/testing/code_coverage.md#view-code-coverage-results-in-the-mr) | Code coverage report for the merge request indicates a reduction in coverage of between 1% to 5%. |
| Code coverage down less than 1% | Info | [Code coverage](../../../ci/testing/code_coverage.md#view-code-coverage-results-in-the-mr) | Code coverage report for the merge request indicates a reduction in coverage of less than 1%. |
<!-- vale gitlab.SubstitutionWarning = YES -->
......
......@@ -42,7 +42,7 @@ To see the latest code coverage for each project in your group:
1. In the **Latest test coverage results** section, from the **Select projects** dropdown list, choose the projects you want to check.
You can download code coverage data for specific projects using
[code coverage history](../../../ci/pipelines/settings.md#view-code-coverage-history).
[code coverage history](../../../ci/testing/code_coverage.md#view-history-of-project-code-coverage).
## Download historic test coverage data
......
......@@ -62,7 +62,7 @@ You can access a test coverage report badge image by using the following link:
https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg
```
You can define the regular expression for the [coverage report](../../ci/pipelines/settings.md#merge-request-test-coverage-results)
You can define the regular expression for the [coverage report](../../ci/testing/code_coverage.md#view-code-coverage-results-in-the-mr)
that each job log is matched against.
This means that each job in the pipeline can have the test coverage percentage value defined.
......
......@@ -101,7 +101,7 @@ Without the approvals, the work cannot merge. Required approvals enable multiple
database, for all proposed code changes.
- Use the [code owners of changed files](rules.md#code-owners-as-eligible-approvers),
to determine who should review the work.
- Require an [approval before merging code that causes test coverage to decline](../../../../ci/pipelines/settings.md#coverage-check-approval-rule)
- Require an [approval before merging code that causes test coverage to decline](../../../../ci/testing/code_coverage.md#coverage-check-approval-rule)
- Users on GitLab Ultimate can also [require approval from a security team](../../../application_security/index.md#security-approvals-in-merge-requests)
before merging code that could introduce a vulnerability.
......
......@@ -244,6 +244,28 @@ approval rule for certain branches:
1. To enable this configuration, read
[Code Owner's approvals for protected branches](../../protected_branches.md#require-code-owner-approval-on-a-protected-branch).
## Code coverage check approval
You can require specific approvals in the case that a merge request would reduce code test
coverage.
For more information, see [Coverage check approval rule](../../../../ci/testing/code_coverage.md#coverage-check-approval-rule).
## Security Approvals **(ULTIMATE)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357021) in GitLab 15.0.
You can use [scan result policies](../../../application_security/policies/scan-result-policies.md#scan-result-policy-editor) to define security approvals based on the status of vulnerabilities in the merge request and the default branch.
Details for each security policy is shown in the Security Approvals section of your Merge Request configuration.
The security approval rules are applied to all merge requests until the pipeline is complete. The application of the
security approval rules prevents users from merging in code before the security scans run. After the pipeline is
complete, the security approval rules are checked to determine if the security approvals are still required.
![Security Approvals](img/security_approvals_v15_0.png)
These policies are both created and edited in the [security policy editor](../../../application_security/policies/index.md#policy-editor).
## Troubleshooting
### Approval rule name can't be blank
......@@ -265,18 +287,3 @@ isn't recognized as a valid Code Owner for the project, nor does it display in t
project's **Approvals** list. To fix this problem, add the approval group as a shared group
high enough in the shared hierarchy so the project requiring review inherits this
group of users.
## Security Approvals **(ULTIMATE)**
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357021) in GitLab 15.0.
You can use [scan result policies](../../../application_security/policies/scan-result-policies.md#scan-result-policy-editor) to define security approvals based on the status of vulnerabilities in the merge request and the default branch.
Details for each security policy is shown in the Security Approvals section of your Merge Request configuration.
The security approval rules are applied to all merge requests until the pipeline is complete. The application of the
security approval rules prevents users from merging in code before the security scans run. Once the pipeline is
complete, the security approval rules are checked to determine if the security approvals are still required.
![Security Approvals](img/security_approvals_v15_0.png)
These policies are both created and edited in the [security policy editor](../../../application_security/policies/index.md#policy-editor).
......@@ -153,13 +153,6 @@ To do this:
select **Remove approvals by Code Owners if their files changed**.
1. Select **Save changes**.
## Code coverage check approvals
You can require specific approvals if a merge request would result in a decline in code test
coverage.
For more information, see [Coverage check approval rule](../../../../ci/pipelines/settings.md#coverage-check-approval-rule).
## Settings cascading
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) in GitLab 14.4. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../../../administration/feature_flags.md), disabled by default.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment