Skip to content

Allow to define the desired concurrency level for a given workflow or group of workflows

Current situation

...

Desired outcome

concurrency

Use concurrency to ensure that only a single job or workflow using the same concurrency group will run at a time. A concurrency group can be any string or expression. The expression can only use opentf, inputs, and variables contexts. For more information about expressions, see "Expressions."

You can also specify concurrency at the job level. For more information, see jobs.<job_id>.concurrency.

When a concurrent job or workflow is queued, if another job or workflow run using the same concurrency group is in progress, the queued job or workflow run will be pending. Any pending job or workflow run in the same concurrency group will be canceled when a new job or workflow run is queued. To also cancel any currently running job or workflow run in the same concurrency group, specify cancel-in-progress: true.

Notes:

  • The concurrency group name is case insensitive. For example, prod and Prod will be treated as the same concurrency group.
  • Ordering is not guaranteed for jobs or workflow runs using concurrency groups. Jobs or workflow runs in the same concurrency group are handled in an arbitrary order.

Example: Using concurrency and the default behavior

The default behavior of the OpenTestFactory orchestrator is to allow multiple jobs or workflow runs to run concurrently. The concurrency keyword allows you to control the concurrency of workflow runs.

For example, you can use the concurrency keyword immediately after where trigger conditions are defined to limit the concurrency of entire workflow runs for a specific branch:

metadata:
  name: ...
concurrency:
  group: ${{ opentf.workflow }}-${{ variables.ref }}
  cancel-in-progress: true

You can also limit the concurrency of jobs within a workflow by using the concurrency keyword at the job level:

metadata:
  name: ...
jobs:
  job-1:
    runs-on: ubuntu-latest
    concurrency:
      group: example-group
      cancel-in-progress: true

Example: Concurrency groups

Concurrency groups provide a way to manage and limit the execution of workflow runs or jobs that share the same concurrency key.

The concurrency key is used to group workflows or jobs together into a concurrency group. When you define a concurrency key, the orchestrator ensures that only one workflow or job with that key runs at any given time. If a new workflow run or job starts with the same concurrency key, the orchestrator will cancel any workflow or job already running with that key. The concurrency key can be a hard-coded string, or it can be a dynamic expression that includes context variables.

It is possible to define concurrency conditions in your workflow so that the workflow or job is part of a concurrency group.

This means that when a workflow run or job starts, the orchestrator will cancel any workflow runs or jobs that are already in progress in the same concurrency group. This is useful in scenarios where you want to prevent parallel runs for a certain set of a workflows or jobs, such as the ones used for deployments to a staging environment, in order to prevent actions that could cause conflicts or consume more resources than necessary.

In this example, job-1 is part of a concurrency group named staging_environment. This means that if a new run of job-1 is triggered, any runs of the same job in the staging_environment concurrency group that are already in progress will be cancelled.

jobs:
  job-1:
    runs-on: ubuntu-latest
    concurrency:
      group: staging_environment
      cancel-in-progress: true

Alternatively, using a dynamic expression such as concurrency: ci-${{ variables.ref }} in your workflow means that the workflow or job would be part of a concurrency group named ci- followed by the reference of the branch or tag that triggered the workflow. In this example, if a new commit is pushed to the main branch while a previous run is still in progress, the previous run will be cancelled and the new one will start:

concurrency:
  group: ci-${{ variables.ref }}
  cancel-in-progress: true

Example: Using concurrency to cancel any in-progress job or run

To use concurrency to cancel any in-progress job or run in the orchestrator, you can use the concurrency key with the cancel-in-progress option set to true:

concurrency:
  group: ${{ variables.ref }}
  cancel-in-progress: true

Note that in this example, without defining a particular concurrency group, the orchestrator will cancel any in-progress run of the job or workflow.