Test Coverage visualization even if of manual jobs do not finish

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Proposal

Currently test coverage visualization only displays after the pipeline is complete. If the pipeline has a blocking manual job, the pipeline waits for the manual job before continuing and is not considered complete. The visualization cannot be displayed if the blocking manual job did not run.(https://docs.gitlab.com/ee/ci/testing/test_coverage_visualization.html#limits)

This proposal is for visualization to be displayed even if manual jobs do not complete.

Use case

Customer apps have the following flow:

  1. MR submitted
  2. Build/tests run
  3. Code Review
  4. Code changes based on reviews (and then more building/testing)
  5. Approval
  6. Manual job kicked off, which are our End to End (E2E) tests
  7. Push the Merge after pipeline finishes button
  8. Eventually it gets merged (assuming the E2E tests pass)

Why the E2E tests are run after after everything: they take a long time to run (~20-25 minutes)

Why the E2E tests are manual jobs: they take up a lot of resources in a shared Kubernetes cluster and if we have too many going at once we have resource contention problems. Those contention problems were solved after making it manual (because many less jobs were being run)

In this flow, they never see the coverage visualization because the manual job runs last and only after it completes do we get the full picture of coverage, which makes it a little useless (at least during the MR flow)

Note that the E2E tests don't have any coverage reports coming out of them, so the percentages/lines wouldn't be any different if the E2E jobs run or not.

Requesting customers:

Questions

  1. If the manual jobs don't have the coverage keyword, shouldn't it be safe to assume the coverage reports have been fully processed and can therefore be displayed?
Edited by 🤖 GitLab Bot 🤖