Skip to content

GitLab Next

Open
Created by Fabio Pitino@fabiopitino🔴Maintainer

Code coverage does not include data from child pipeline jobs

Summary

Suppose to have a parent-child pipeline configuration where a child pipeline contains jobs that output code coverage status.

# .gitlab-ci.yml -- parent pipeline
rspec:
  stage: test
  script:
    - echo "running rspec..."
    - echo "Code coverage 80"
  coverage: /Code coverage \d+/

trigger-tests:
  stage: test
  trigger:
    include: child.yml
    strategy: depend
# child.yml -- child pipeline
rspec:
  script:
    - echo "Running tests..."
    - echo "Code coverage 40"
  coverage: /Code coverage \d+/

karma:
  script:
    - echo "running karma tests..."
    - echo "Code coverage 30"
  coverage: /Code coverage \d+/

Maybe a more realistic scenario would be to have karma tests to run in a dedicated child pipeline and rspec tests to run in a dedicated child pipeline. Both coverage reports would be expected to be visible from the parent pipeline.

In this example, I purposely added a rspec job in the parent pipeline to demonstrate that the coverage from the 2 rspec jobs should be averaged because both appear to be part of the same job group, even though across 2 different pipelines.

Problem 1: MR widget

When looking at the MR widget that shows the code coverage, it does not take in consideration jobs from child pipelines. Coverage from child pipelines is never surfaced in the MR widget.

image

The widget shows 1 job reporting rspec coverage 80% while it should be a 60% (80 + 40 / 2) over 2 jobs. It's also missing the karma coverage.

Problem 2: Daily code coverage report analytics

When looking at the coverage analytics (Analytics > Repository), the code coverage does not show data from child pipelines either.

The data should show 60% for rspec coverage and 30% for karma.

NOTE If the child pipeline is triggered without strategy:depend it means that the parent pipeline will not wait for it to finish and the bridge job immediately succeeds. For these types of pipelines we can't guarantee that we have a consistent coverage report because the parent may finish before the child pipeline. We should consider coverage jobs only from dependent child pipelines (using strategy:depend).

What is the current bug behavior?

The child pipeline's coverage are not considered by the parent pipeline. The purpose of child pipelines is to compose the parent pipeline. This means doing tasks on behalf of the parent pipeline while running them in isolation.

Example project: https://gitlab.com/Lonli-Lokli/cv-app

What is the expected correct behavior?

Code coverage should report data from all coverage jobs in the pipeline hierarchy.

Possible fixes

This MR !47200 (closed) aims to fix the coverage data shown in the Analytics > Repository.

The MR explicitly depends on !46576 (closed) to be merged first, which provides the ability to list jobs from dependent parent pipelines.

Documentation

If !51516 (merged) is merged, we should remove that line when this issue is completed.

Edited by Ricky Wiens