Skip to content

resource_group is not correctly limiting seperate pipelines from running jobs interleaved between the other. Pipelines should finish before another starts.

Summary

Resource groups should prevent two pipelines from utilising the same resource until the first pipeline completes or has failed. in my circumstance, this doesn't appear to be the case. Two pipelines with the same resource group can end up running interleaved (pipe1-job1, pipe2-job1, pipe1-job2 etc), which will break a dev deployment resource, requring stages running in order cleanly for any given pipe.

Steps to reproduce

This is an example of the template that I use in all the jobs -

.job_template_dev: &job_definition_dev
  resource_group: dev_resource
  dependencies: [] # we specify no dependencies so that artifacts dont all get aquired (default), which would orphan the tfstate file.
  rules:
    - if: $single_stage == 'disabled' && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: always
      allow_failure: false
    - if: $single_stage == $CI_JOB_STAGE && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: always
      allow_failure: true
    - if: $destroy_on_failure == 'true' && $CI_JOB_STAGE == 'cleanup_terraform_destroy_dev' && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: on_failure
      allow_failure: true
    - if: $manual == 'true' && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: manual
      allow_failure: true
    - if: $single_stage != $CI_JOB_STAGE && $CI_COMMIT_BRANCH != "master" && $CI_COMMIT_BRANCH != "stage"
      when: manual
      allow_failure: true
  tags: # the runner is tagged dev
    - dev

-The idea here is we generally use the top rule for most jobs. -Sometimes the single_stage var is use to run only one stage in the pipe for faster iteration over a problematic test. -If infrastructure rollout fails we have a destroy job that can cleanup, and is run at the end. -Some jobs can be run manually. -When we run single jobs, its also possible to manually trigger other jobs as well.

This screenshot demonstrates two jobs running interleaved on the same resource which should be avoided.

Screenshot_from_2020-04-20_22-44-57

The gitlab server is running 12.8.7

and gitlab runner: gitlab-runner --version Version: 12.9.0 Git revision: 4c96e5ad Git branch: 12-9-stable GO version: go1.13.8 Built: 2020-03-20T13:01:56+0000 OS/Arch: linux/amd64

What is the current bug behavior?

Two pipes can endup running interleaved jobs on the same resource.

What is the expected correct behavior?

Pipelines should finish or error before a second pipeline will run.

Edited by Andrew Graham