Limit pipeline concurrency using named semaphores
This issue was moved (and resolved) in another project.
Problem to Solve
Some pipelines and/or jobs use unique resources or are in some way destructive to an environment. Being able to limit concurrency for them would allow users control over scenarios where there should only be one deploy at a time for an:
- Environment
- Entire Project
- Job (perhaps due to shared testing infrastructure in a testing lab)
Solution
We will have the syntax lock: semaphore-name
which could be applied at a job or pipeline level, and would create a project-level semaphore preventing jobs or pipelines from running which also claim the same semaphore.
- Pipelines/jobs would wait for it to become available, and time out after some point (probably just based on pipeline time out).
- Locked jobs are guaranteed to be executed in the order created by pipeline, so the job would receive a
blocked
status to indicate that it waits for being unlocked by anotherpending/running
job.
Implicit locking for environments
Because environments are much more often than not the kind of place where you'd want only one deployment to run at once, and always in the correct order, we will include implicit locking wherever environment:
is used, using a semaphore with the name of the environment.
- When
environment:
is used, it implieslock:
, so you don't need to specifylock:
andenvironment:
, - When
environment:
is used, you can uselock: some-name
to create a lock across all environment deployments, - When implict lock is used, you can define
lock: nil
to disable locking, thus run with full concurrency limit, - Implicit lock for the environment comes from the assumption that all deployments are by design not working very well when executed concurrently
Sample Configuration
This example will run only ever one of the project's pipeline's at once. The pipeline itself will run as normal, with all jobs running in parallel in the build stage.
lock: $CI_PROJECT_NAME
# lock: $CI_ENVIRONMENT_NAME for example would give you a way to run one entire pipeline per environment
stages:
- build
jobA:
stage: build
script:
- echo HelloA
jobB:
stage: build
script:
- echo HelloB
This example moves the lock to a job. Multiple pipelines can run simultaneously, but jobA
will only ever run one at a time, across all pipelines in the project.
stages:
- build
jobA:
lock: jobA
stage: build
script:
- echo HelloA
jobB:
stage: build
script:
- echo HelloB
Future Improvements
Different concurrency behaviors
At the moment, all this will do is wait for a semaphore to free up. You could imagine more possibilities:
concurrency:
parallel: Default current value, job is launch even if an other is in progress
cancel : Cancel job if is launch in parallel of another
wait: Wait previous job is finish for launch current
skip: Skips job, if lock is already acquired