Upgrade undercover to 0.9.0 and compare commits in CI

What does this MR do and why?

scripts/undercoverage (the undercoverage check, which runs in rspec:coverage since !265673 (merged)) takes from about 10s to several minutes in CI, depending on the merge request. In 30 rspec:undercoverage jobs from tier 3 MR pipelines on 2026-10-26 and 2026-10-27, it took a median of 12s, but 27–113s in 6 of them.

undercover 0.8.5 had two problems:

  1. It diffed the compare base against the index and the working tree. That reads every file in the checkout (about 115,000), as the git diff in !262134 (merged) did: jobs restore the checkout from the clone-gitlab-repo artifact, so the file metadata in the index doesn't match the files on disk.
  2. It looked up each file's coverage with a linear search over the about 25,000 files in the report, normalizing the path again for each of them, for every method and block of each changed file. Changes to big files took minutes.

undercover 0.9.0 looks coverage up directly, matches changed lines to code in linear time (#274), and adds --commits-only (#273), which compares the commits only. This MR:

  • Upgrades undercover from 0.8.5 to 0.9.0. Its new dependencies (prism, erubi) are already in the bundle, and imagen is no longer needed.
  • Passes --commits-only in CI, where the checkout is the commit without local changes. Locally, undercover still checks uncommitted changes too.
  • Passes --no-hint, to keep 0.9.0's new hint about UndercoverCI out of the output.
  • Removes the Result#pretty_print patch. It made undercover show branch coverage on lines without line coverage, which undercover does itself since 0.8.4 (#249). The compare_base_obj patch for shallow clones stays.
  • Stops printing git diff --stat before the check. It only printed the diff, and for the same reason as the first problem above it took 40s in !265673 (merged)'s pipeline.

When an MR only changes files that undercover excludes (for example specs, or .gitlab/ci), 0.9.0 prints WARNING: nothing was checked - all N candidate file(s) were skipped by --include-files/--exclude-files and passes, as 0.8.5 did without the warning.

Results

In job 17528116872 of this MR's tier 3 pipeline, a temporary commit ran this branch's scripts/undercoverage (0.9.0) and master's (0.8.5, installed next to the bundle) against the same coverage report, for three compare bases:

Compare base Changed .rb files 0.9.0 0.8.5 0.9.0 again Output
This MR's target branch commit 0 2.0s 10.2s 2.0s identical, exit code 0
3 commits before it on master 13 2.1s 95.9s 2.1s identical, exit code 0
10 commits before it on master 42 2.2s 183.3s 2.4s identical, exit code 1 (one method flagged in ee/app/services/concerns/secrets_management/secrets/read_metadata_service_helpers.rb)

The temporary commit is removed from this branch.

With !265542 (merged) and !265673 (merged) merged, this MR's tier 3 pipeline 3005882647 compares with the medians of 113 tier 3 MR pipelines from 2026-10-26 and 2026-10-27, before any of these changes:

Before (median) !265673's pipeline This MR (job 17589845884)
git diff --stat 7s 40s removed
undercover, with bundle exec 12s 15s 4.3s
Last RSpec job to the end of the coverage tail 371s 328s 276s

That's about 95s off the tail, from one pipeline.

How to set up and validate locally

  1. Download the coverage/lcov/gitlab.lcov artifact of a recent rspec:coverage job into coverage/lcov/gitlab.lcov, and also as coverage/lcov/<name of your checkout directory>.lcov. touch both so they're newer than your changes.
  2. Commit a change that adds a method without specs, for example in lib/gitlab/access.rb.
  3. Run time CI=true bundle exec scripts/undercoverage <commit before your change> on this branch, and on master.
  4. Both report the new method and exit with 1. On this branch it takes a few seconds.
  5. Without CI, an uncommitted change is reported too.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Heinrich Lee Yu

Merge request reports

Loading
Loading