Code coverage does not include data from child pipeline jobs
Summary
Suppose to have a parent-child pipeline configuration where a child pipeline contains jobs that output code coverage status.
# .gitlab-ci.yml -- parent pipeline
rspec:
stage: test
script:
- echo "running rspec..."
- echo "Code coverage 80"
coverage: /Code coverage \d+/
trigger-tests:
stage: test
trigger:
include: child.yml
strategy: depend
# child.yml -- child pipeline
rspec:
script:
- echo "Running tests..."
- echo "Code coverage 40"
coverage: /Code coverage \d+/
karma:
script:
- echo "running karma tests..."
- echo "Code coverage 30"
coverage: /Code coverage \d+/
Maybe a more realistic scenario would be to have karma
tests to run in a dedicated child pipeline and rspec
tests to run in a dedicated child pipeline. Both coverage reports would be expected to be visible from the parent pipeline.
In this example, I purposely added a rspec
job in the parent pipeline to demonstrate that the coverage from the 2 rspec
jobs should be averaged because both appear to be part of the same job group, even though across 2 different pipelines.
Problem 1: MR widget
When looking at the MR widget that shows the code coverage, it does not take in consideration jobs from child pipelines. Coverage from child pipelines is never surfaced in the MR widget.
The widget shows 1 job reporting rspec
coverage 80% while it should be a 60% (80 + 40 / 2) over 2 jobs. It's also missing the karma
coverage.
Problem 2: Daily code coverage report analytics
When looking at the coverage analytics (Analytics > Repository
), the code coverage does not show data from child pipelines either.
The data should show 60% for rspec coverage and 30% for karma.
NOTE If the child pipeline is triggered without strategy:depend
it means that the parent pipeline will not wait for it to finish and the bridge job immediately succeeds. For these types of pipelines we can't guarantee that we have a consistent coverage report because the parent may finish before the child pipeline. We should consider coverage jobs only from dependent child pipelines (using strategy:depend
).
What is the current bug behavior?
The child pipeline's coverage are not considered by the parent pipeline. The purpose of child pipelines is to compose the parent pipeline. This means doing tasks on behalf of the parent pipeline while running them in isolation.
Example project: https://gitlab.com/Lonli-Lokli/cv-app
What is the expected correct behavior?
Code coverage should report data from all coverage jobs in the pipeline hierarchy.
Possible fixes
This MR !47200 (closed) aims to fix the coverage data shown in the Analytics > Repository
.
The MR explicitly depends on !46576 (closed) to be merged first, which provides the ability to list jobs from dependent parent pipelines.
Documentation
If !51516 (merged) is merged, we should remove that line when this issue is completed.