Support for multiple resource groups per job (Multiple exclusive locks)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem to solve

Currently, a single job can be attached to only one resource group (i.e. single exclusive lock key). To illustrate:

test:
  ...
  resource_group: Test servers

deploy:
  ...
  resource_group: Deployment servers

This means that test job still can run when deploy job is running, however, in some cases, users want prevent test job from running while deploy job is running. So attaching multiple resource groups to a single job helps this situation:

test:
  ...
  resource_group:
    - Test servers
    - Deployment servers

deploy:
  ...
  resource_group: Deployment servers

This way, test job requires a resource from each Test servers and Deployment servers.

Proposal

Support the specification of multiple available resources to the resource_group and the jobs for which those resources can be applied to.

shared_resources:
  - databases:
    - db1
    - db2
    - ...
  - noSqlDatabases:
    - nosql1
    - nosql2
    - ...

integrationTest:
  resource_group:
    - databases
    - databases
    - noSqlDatabases
  script:
    - echo $booked_database # any from shared_resources.databases but used by 1 job only
    - echo $booked_database_1 # any from shared_resources.databases but used by 1 job only (different than above)
    - echo $booked_noSqlDatabases # any from shared_resources.noSqlDatabases


OR


integrationTest:
  resource_group:
    - databases
    - databases
    - noSqlDatabases
  script:
    - echo $resources[0] # any from shared_resources.databases but used by 1 job only
    - echo $resources[1] # any from shared_resources.databases but used by 1 job only (different than above)
    - echo $resources[2] # any from shared_resources.noSqlDatabases



LESS efficient in terms of rotating resources, but more redeable:

shared_resources:
  - databases:
    - db1
    - db2
    - ...
  - databases_downstream:
    - db1
    - db2
    - ...
  - noSqlDatabases:
    - nosql1
    - nosql2
    - ...

integrationTest:
  resource_group:
    - databases
    - databases_downstream
    - noSqlDatabases
  script:
    - echo $database # any available from shared_resources.databases, but constant for whole job
    - echo $databases_downstream  # could be tricky to remove s
    - echo $noSqlDatabase

Example Scenario

Copied from #227589 (closed)

Here is a scenario where wait for multiple resource_group should help:

  • Deployment job into container based environments (like Kubernetes) can support multiple deployments in parallel. In this case the resource group is required to ensure that only single deployment happens for a given service in an environment. Creating a per service resource group allows the deployment of different services in parallel while preventing multiple deployment of the same service.
  • Now apart from preventing multiple deployments of the same service I also want to prevent deployment if an acceptance test job is running in that environment.

This is the situation when wait on multiple resource_group show allow to wait for any job deploying the same service and wait while acceptance test job is running.

Further details

Keep in mind that this issue is not for increasing concurrency.

For the concurrency increase, please see Support for multiple resources per resource group (Concurrency control)

Edited by 🤖 GitLab Bot 🤖