Skip to content

Resource group that spans multiple jobs in a pipeline

Release notes

Problem to solve

Currently, resource_groups protect from running two jobs with a common resource_group at the same time.

There needs to be an analagous handling for multiple jobs in a single pipeline that share a resource group.

Use case: multi-stage docker images

For efficiency, you split building each stage of a multi-stage docker image into a job.

However, the build cache tags are static strings like :stage1, :stage2, :stage3 so that the build cache for each stage is overwritten on each commit.

For example, in the following gitlab.yml, the jobs in pipeline1 all share the same resource and need to access it with parallel execution:

build-image-base:
  extends: .build-image-settings
  stage: build-base
  script:
    # base image
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_BASE_IMAGE || echo Build cache skipped or unavailable'
    - docker build $([[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && echo "--cache-from $TAGGED_CI_CACHE_BASE_IMAGE" || echo "--no-cache") --target base --tag $TAGGED_CI_CACHE_BASE_IMAGE --build-arg BUILDKIT_INLINE_CACHE=1 .
    - docker push $TAGGED_CI_CACHE_BASE_IMAGE

build-image-stage1:
  extends: .build-image-settings
  stage: build-stages
  script:
    # stage1 image
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_BASE_IMAGE || echo Build cache skipped or unavailable'
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_STAGE1_IMAGE || echo Build cache skipped or unavailable'
    - docker build --cache-from $TAGGED_CI_CACHE_BASE_IMAGE $([[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && echo "--cache-from $TAGGED_CI_CACHE_STAGE1_IMAGE" || echo "") --target stage1 --tag $TAGGED_CI_CACHE_STAGE1_IMAGE --build-arg BUILDKIT_INLINE_CACHE=1 .
    - docker push $TAGGED_CI_CACHE_STAGE1_IMAGE

build-image-stage2:
  extends: .build-image-settings
  stage: build-stages
  script:
    # stage2 image
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_BASE_IMAGE || echo Build cache skipped or unavailable'
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_STAGE2_IMAGE || echo Build cache skipped or unavailable'
    - docker build --cache-from $TAGGED_CI_CACHE_BASE_IMAGE $([[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && echo "--cache-from $TAGGED_CI_CACHE_STAGE2_IMAGE" || echo "") --target stage2 --tag $TAGGED_CI_CACHE_STAGE2_IMAGE --build-arg BUILDKIT_INLINE_CACHE=1 .
    - docker push $TAGGED_CI_CACHE_STAGE2_IMAGE

build-image:
  extends: .build-image-settings
  stage: build
  script:
    # untargeted image
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_STAGE1_IMAGE || echo Build cache skipped or unavailable'
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_STAGE2_IMAGE || echo Build cache skipped or unavailable'
    - '[[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && docker pull $TAGGED_CI_CACHE_UNTARGETED_IMAGE || echo Build cache skipped or unavailable'
    - docker build --cache-from $TAGGED_CI_CACHE_STAGE2_IMAGE --cache-from $TAGGED_CI_CACHE_STAGE1_IMAGE $([[ $DOCKER_CACHE_FROM_LATEST -eq "1" ]] && echo "--cache-from $TAGGED_CI_CACHE_UNTARGETED_IMAGE" || echo "") --tag $TAGGED_CI_CACHE_UNTARGETED_IMAGE --tag $TAGGED_CI_REGISTRY_IMAGE --build-arg BUILDKIT_INLINE_CACHE=1 .
    - docker push $TAGGED_CI_CACHE_UNTARGETED_IMAGE
    - docker push $TAGGED_CI_REGISTRY_IMAGE

However, until all jobs in pipeline1 that use the resource complete, pipeline2 should not be able to access it. The currently available 'resource_groups' option does not guarantee this. When you have two pipelines competing for the resource shared among the jobs it will ping pong back and forth between jobs in pipeline1 and pipeline2 thus corrupting the build cache between the jobs of both pipelines.

If you add this "shared_resource_group" and allow it to be used alongside "resource_group" then you can also support multi-job resource_groups by specifying both at the same time.

I.e. provide exclusive use to the resource for a pipeline and each job in that pipeline has exclusive use. When used individually the pipeline would have exclusive use but the jobs would have simultaneous use.

This would be of tremendous value for building multi-stage docker images. Plus other cloud deployment scenarios.

Use case: multi-step deployment

Complex deployments (e.g. gitlab.com) often require multiple steps to roll out. In these situations, it would be beneficial to have a whole pipeline lock a resource group and have the jobs in sequence within a pipeline, and multiple pipelines would queue up.

An alternative approach seems to use merge trains for similar scenarios, but merge trains typically slow down these setups a lot, as they are already complex to start with.

Proposal

Define a global resource_group CI syntax that applies to all the jobs within the pipeline and requires the pipeline to finish to unlock the resource group for jobs from other pipelines.

Intended users

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by 🤖 GitLab Bot 🤖