Add code coverage badges to Dependency Scanning projects (Go)
Proposal
In the Dependency Scanning analyzer projects, add badges that measure the code coverage of Go code, when running the unit tests.
Further details
Instructions to measure code coverage when running Go unit tests:
go test -coverprofile cover.out -v ./...
go tool cover -func cover.out
The output of go tool cover
gives the code coverage of each Go package being tested. The last line gives the total
coverage score.
See example in gemnasium-maven
% go test -v -coverprofile=cover.out ./...
? gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2 [no test files]
? gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder [no test files]
=== RUN TestExtract
=== RUN TestExtract/ExtractGradle
=== RUN TestExtract/ExtractMaven
=== RUN TestExtract/ExtractSbt
--- PASS: TestExtract (0.00s)
--- PASS: TestExtract/ExtractGradle (0.00s)
--- PASS: TestExtract/ExtractMaven (0.00s)
--- PASS: TestExtract/ExtractSbt (0.00s)
=== RUN TestSplit
=== RUN TestSplit/with_root_path
=== RUN TestSplit/without_root_path
--- PASS: TestSplit (0.00s)
--- PASS: TestSplit/with_root_path (0.00s)
--- PASS: TestSplit/without_root_path (0.00s)
PASS
coverage: 100.0% of statements
ok gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/exportpath 1.759s coverage: 100.0% of statements
? gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/gradle [no test files]
? gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/maven [no test files]
? gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/sbt [no test files]
=== RUN TestImportCmd
=== RUN TestImportCmd/java8
=== RUN TestImportCmd/java11
=== RUN TestImportCmd/java13
=== RUN TestImportCmd/java14
--- PASS: TestImportCmd (0.00s)
--- PASS: TestImportCmd/java8 (0.00s)
--- PASS: TestImportCmd/java11 (0.00s)
--- PASS: TestImportCmd/java13 (0.00s)
--- PASS: TestImportCmd/java14 (0.00s)
PASS
coverage: 42.9% of statements
ok gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/keystore 1.843s coverage: 42.9% of statements
=== RUN TestReportScanner
--- PASS: TestReportScanner (0.00s)
PASS
coverage: [no statements]
ok gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/metadata 0.928s coverage: [no statements]
? gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/plugin [no test files]
% go tool cover --func=cover.out
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/exportpath/exportpath.go:12: ExtractGradle 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/exportpath/exportpath.go:17: ExtractMaven 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/exportpath/exportpath.go:22: ExtractSbt 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/exportpath/exportpath.go:30: extract 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/builder/exportpath/exportpath.go:50: Split 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/keystore/keystore.go:18: Update 0.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/keystore/keystore.go:30: importCmd 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/keystore/keystore.go:43: keystorePathFor 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/v2/keystore/keystore.go:50: isJava8 100.0%
total: (statements) 77.1%
main
package
By default the main
package is NOT included in code coverage. This can be changed by setting --coverpkg
to <Go module name>/...
. This can be achieve in a generic way using $(go list .)/...
.
go tool cover --func=cover.out|grep total
go test -v -coverprofile=cover.out --coverpkg=$(go list .)/... ./...
(In the case of gemnasium
, the coverage score when from 80.60% to 58.1% because the main
package is pretty big compared to the other packages.)
git submodules
If --coverpkg
is set to ./...
, then all Go packages are tested, including the ones that are git submodule of the git repo being tested. In the case of gemnasium-maven
and gemnasium-python
, this would result in considering the gemnasium
git submodule when measuring code coverage.
See example in gemnasium-python
% go test -v -coverprofile=cover.out --coverpkg=$(go list .)/... ./...
? gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2 [no test files]
=== RUN TestReportScanner
--- PASS: TestReportScanner (0.00s)
PASS
coverage: 1.0% of statements in gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/...
ok gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/metadata 0.957s coverage: 1.0% of statements in gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/...
=== RUN TestMatch
=== RUN TestMatch/0
=== RUN TestMatch/1
=== RUN TestMatch/2
=== RUN TestMatch/3
=== RUN TestMatch/4
=== RUN TestMatch/5
=== RUN TestMatch/6
=== RUN TestMatch/7
=== RUN TestMatch/8
--- PASS: TestMatch (0.00s)
--- PASS: TestMatch/0 (0.00s)
--- PASS: TestMatch/1 (0.00s)
--- PASS: TestMatch/2 (0.00s)
--- PASS: TestMatch/3 (0.00s)
--- PASS: TestMatch/4 (0.00s)
--- PASS: TestMatch/5 (0.00s)
--- PASS: TestMatch/6 (0.00s)
--- PASS: TestMatch/7 (0.00s)
--- PASS: TestMatch/8 (0.00s)
PASS
coverage: 9.3% of statements in gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/...
ok gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/plugin 0.957s coverage: 9.3% of statements in gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/...
% go tool cover --func=cover.out
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/main.go:43: main 0.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/main.go:63: findCommand 0.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/main.go:98: runCommand 0.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/plugin/plugin.go:13: Match 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/v2/plugin/plugin.go:29: init 100.0%
total: (statements) 9.3%
Adding badges to projects
Badges can be added manually via the general settings page of the project. GitLab automatically generated an SVG badge based on the code coverage collected in the project pipeline. Badge URL is: https://gitlab.com/%{project_path}/badges/%{default_branch}/badge.svg
.
Cobertura export
The generated file cover.out
can then be converted to Cobertura XML using the gocover-cobertura command, and upload as a cobertura artifact. Cobertura artifacts are used for test coverage visualization in the MR.
This is out of scope.
Proposal
- Exclude the
main
package from code coverage. This should be measured when running integration tests though. - Excluded git submodule from code coverage.
- Add the coverage badge for the default branch, and make it link to the commits page for that branch:
https://gitlab.com/%{project_path}/-/commits/%{default_branch}
.
Implementation plan
- Check the job definition of
go test
(shared CI config), and make sure that it collects the overall code coverage score. - Add
coverage.svg
badges to all Dependency Scanning projects.
Outcome
See badges of the gemnasium project:
See go test
where this metric was collected:
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/vrange/golang/semver.go:288: isBadNum 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/vrange/golang/semver.go:296: isNum 100.0%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/vrange/golang/semver.go:304: compareInt 77.8%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/vrange/golang/semver.go:320: comparePrerelease 73.3%
gitlab.com/gitlab-org/security-products/analyzers/gemnasium/v2/vrange/golang/semver.go:379: nextIdent 100.0%
total: (statements) 80.6%