ci: gate test coverage of an MR's changed Go lines

Summary

A new job, test:changed-line-coverage, fails an MR when tests cover less than 80% of the Go lines it changes. It counts changed lines only, not the package or the repo. Generated files and test code do not count: _test.go files, the test-support packages, and e2e/.

A changed line is covered when any test job in the same pipeline ran it:

  • go_unittests, which already writes a coverage profile.
  • The Go integration jobs, which run when their code changes: test:integration, the datastore and migrations suites, the three Redis jobs, test:accounting-failpoints, test:integration:s3-garage, test:integration:conformance:s3-garage and test:integration:gcs-key-creds. Each now keeps its raw profile as coverage/raw-<job>.out.

scripts/ci/check-changed-line-coverage.sh does the counting. The changed-line-coverage pre-commit hook runs the same script against origin/main.

Why integration coverage counts too

Store code builds SQL and only does work against Postgres. Its tests are integration-tagged, so the unit run never executes it. A unit-only gate scores those lines at 0, even when the MR adds tests for them. Pipelines must pass before merge and the gate has no override, so unit-only would block datastore MRs that do test their change.

I replayed the gate over recent merged MRs, each against its own pipeline's profiles:

  • 5 of the last 29 MRs with Go changes fail on unit coverage alone.
  • 5 of 10 recent MRs that change internal/datastore fail on unit coverage alone, 4 of them on datastore lines.
  • !3307 (merged) still had its integration profiles: 65/95 lines (68.4%) on unit alone, 91/95 (95.8%) with its datastore and migrations suites. So it fails one way and passes the other.

Costs and gaps

  • The gate waits for the slowest integration job that runs. On !3307 (merged)'s pipeline, go_unittests finished at 12.0 minutes and the slowest integration job at 14.7. On an MR that changes the GCS driver, the gate also waits for the real-GCS job, whose test budget is 35 minutes.
  • Each of those jobs keeps one more artifact, its raw profile. For scale, go_unittests' whole-repo profile is 5.0 MB.
  • test:integration:gcs-key-creds is the one real-cloud job the gate reads. The GCS driver has no emulator job, so only real GCS runs its integration tests. Forks and the security mirror skip that job, so there GCS driver lines count unit coverage only, and a GCS change can fail the gate there while it passes here.
  • The real AWS S3 jobs stay out. The Garage jobs already run the S3 driver's integration tests.
  • e2e/ is not measured. The e2e suites drive a deployed instance and give no Go coverage, so no job could credit those lines.
  • The hook runs unit tests only, so it skips packages with integration-tagged tests and leaves them to CI. No job uses -coverpkg, so a package's coverage comes only from its own tests. Every other package gets the same answer locally as in CI.

Notes

  • .mr-diff-base is a new anchor for the MR's base commit. It repeats the logic lint:comment-caps has inline, because !3339 (merged) deletes that job. !3332 (merged) could reuse the anchor.
  • post-test:go:coverage, from the modular-feature-testing component, checks whole-repo coverage. It stays turned off in .gitlab/ci/common-ci-tasks-patches.yml.

Governing ADRs

None. I checked ADR-023 (code structure and enforcement): it covers package layout and depguard, not test coverage. scripts/adr-freshness.sh reports the mirror one commit behind, on ADR-003, which this MR does not touch.

Testing

  • scripts/ci/check-changed-line-coverage_test.sh has 46 cases. The new test:changed-line-coverage-gate job runs them in the golang CI image. They pass locally.
  • I broke each check in the script one at a time, 28 changes in all. Each one turns the suite red. The hook's two empty-array guards turn it red on bash 3.2 only: CI's bash accepts an empty array under set -u.
  • The CI config lints valid through the GitLab API.

Merge order

None required. This branch merges cleanly with main, !3334 (merged), !3332 (merged) and !3339 (merged).

Closes https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1614

Edited by Dzmitry (Dima) Meshcharakou

Merge request reports

Loading
Loading