Consider Adding rspec:undercoverage in tier-1 pipelines
Problem Statement
Currently, rspec:undercoverage only runs in tier-3 pipelines, providing coverage feedback very late in the development process. By this point, developers have already invested significant time, received approvals, and are mentally "finished" with their MR. Fixing coverage issues requires costly context switching and may delay merges.
Goal
Run rspec:undercoverage analysis using coverage data from tier-1 predictive tests to provide early coverage feedback while developers are actively working on their MRs.
We want to experiment with this approach to evaluate:
- Whether partial coverage data from predictive tests provides actionable feedback
- The false positive rate (lines reported as uncovered that are actually covered by tests not in the prediction)
- Developer adoption and perceived value
Technical Notes
Current Blocker
Child pipeline artifacts: Tier-1 predictive tests run in child pipelines. Historically, accessing artifacts from child pipelines in the parent pipeline has been difficult/impossible, preventing artifact collection and coverage merging.
Key Technical Details
- SimpleCov is enabled by default (
SIMPLECOV: "true") across all CI jobs - Predictive tests generate
.resultset.jsoncoverage files in child pipelines - SimpleCov tracks broadly (controllers, models, services, lib/, etc.)
- Current
scripts/merge-simplecovand artifact collectors only work in tier-3 -
rspec:undercoverageneeds merged coverage file (coverage/lcov/gitlab.lcov)
Next Steps
- Re-evaluate child pipeline artifact access - GitLab may have improved this capability since our last attempt 1-2 years ago
- Prototype with simplified assumptions - Test the approach assuming artifacts can be collected from child pipelines
- Measure value vs. noise - Run parallel tier-1 and tier-3 undercoverage analysis to compare results and false positive rates