Support for multiple resource groups per job (Multiple exclusive locks)
Problem to solve
Currently, a single job can be attached to only one resource group (i.e. single exclusive lock key). To illustrate:
test:
...
resource_group: Test servers
deploy:
...
resource_group: Deployment servers
This means that test
job still can run when deploy
job is running, however, in some cases, users want prevent test
job from running while deploy
job is running. So attaching multiple resource groups to a single job helps this situation:
test:
...
resource_group:
- Test servers
- Deployment servers
deploy:
...
resource_group: Deployment servers
This way, test
job requires a resource from each Test servers
and Deployment servers
.
Proposal
Support the specification of multiple available resources to the resource_group and the jobs for which those resources can be applied to.
shared_resources:
- databases:
- db1
- db2
- ...
- noSqlDatabases:
- nosql1
- nosql2
- ...
integrationTest:
resource_group:
- databases
- databases
- noSqlDatabases
script:
- echo $booked_database # any from shared_resources.databases but used by 1 job only
- echo $booked_database_1 # any from shared_resources.databases but used by 1 job only (different than above)
- echo $booked_noSqlDatabases # any from shared_resources.noSqlDatabases
OR
integrationTest:
resource_group:
- databases
- databases
- noSqlDatabases
script:
- echo $resources[0] # any from shared_resources.databases but used by 1 job only
- echo $resources[1] # any from shared_resources.databases but used by 1 job only (different than above)
- echo $resources[2] # any from shared_resources.noSqlDatabases
LESS efficient in terms of rotating resources, but more redeable:
shared_resources:
- databases:
- db1
- db2
- ...
- databases_downstream:
- db1
- db2
- ...
- noSqlDatabases:
- nosql1
- nosql2
- ...
integrationTest:
resource_group:
- databases
- databases_downstream
- noSqlDatabases
script:
- echo $database # any available from shared_resources.databases, but constant for whole job
- echo $databases_downstream # could be tricky to remove s
- echo $noSqlDatabase
Example Scenario
Copied from #227589 (closed)
Here is a scenario where wait for multiple resource_group should help:
- Deployment job into container based environments (like Kubernetes) can support multiple deployments in parallel. In this case the resource group is required to ensure that only single deployment happens for a given service in an environment. Creating a per service resource group allows the deployment of different services in parallel while preventing multiple deployment of the same service.
- Now apart from preventing multiple deployments of the same service I also want to prevent deployment if an acceptance test job is running in that environment.
This is the situation when wait on multiple resource_group show allow to wait for any job deploying the same service and wait while acceptance test job is running.
Further details
Keep in mind that this issue is not for increasing concurrency.
For the concurrency increase, please see Support for multiple resources per resource group (Concurrency control)