Skip to content

Parallelize CI jobs

Sami Hiltunen requested to merge smh-pipeline-dependencies into master

This MR contains various CI clean ups. Additionally, it enables two features that we haven't been using yet:

  1. GitLab CI supports a feature that allows for explicitly marking job dependencies. This allows for visualizing the job dependency graph and running CI jobs in parallel that do not depend on each other. This MR goes step by step from modeling the existing dependencies to removing those dependencies and allowing all jobs to run in parallel. The graph earlier had bunch of jobs blocking each other, where now we don't have any blocking dependencies between the jobs.

    Noteworthy is that some of the steps that were blocked on build step reused the cache that is created by it. Most of the time this cache is going to be in place as it gets only invalidated by changes to the dependency files and gitlab-ci.yml. In the general case when these files are not modified the cache will be available, and this MR allows for speeding those jobs by removing the unnecessary blocking. The build step takes about 10 minutes so we can see the test results on average 10 minutes faster.

    Pipeline speeds:

    The pipeline speeds above are not 100% precise given these also a lot of variation in the execution times but should give a general idea. The pipeline speed without any caches in place remains as fast as it is before this MR while seeing some of the jobs execute faster as they don't unnecessarily block on the build step.

    Closes #3123 (closed)

  2. GitLab CI supports extracting the test coverage from the test output and displaying it in the MR widget and other places like the coverage history of a repository. The feature is enabled by providing it with a regex that extracts the coverage value. This MR does that, the coverage is visible under the pipeline status.

Edited by Sami Hiltunen

Merge request reports