Group-level resource group
Release notes
Problem to solve
Some pipelines and/or jobs use unique resources or are in some way destructive to an environment.
Born from #15536 (comment 261751575).
Delivering #15536 (closed) would make the lock only applicable in project scope, but there may be multiple projects that use the same
resource_group
. In fact all 3 projects would access same resource at same time.
Intended users
Technical roles (primary, preference towards yaml config):
Non-Technical roles (secondary, preference towards UI):
- QA engineers
- PMs
User experience goal
Being able to limit concurrency for them would allow users control over scenarios where there should only be one deployment at a time for an:
- Environment
- Entire Project
- Group of projects
- Job (perhaps due to shared testing infrastructure in a testing lab)
Further details
Making a queue there it would be great and it should act same as runners - when resource lock is dropped, the next one can take it (hence user-definable number of locks). I like it as per-job lock as that should be the unit in which you expect to operate with that resource and to keep it in equal state.
I know the problem might occur when you have several projects defining different concurrency value, but GitLab server could post a response like: the resource name is used in project XX (with concurrency Y), XX1 (with concurrency Y1), XX2 (with concurrency Y2), XX3 (with concurrency Y3). Using the lowest number of concurrency (Y3) for safety reason.
That would allow us to track per-project defined concurrency values for each resource server wide (or maybe at least group wide, to avoid permission problems). It could also just be a setting in group where you would define resource and concurrency value for that group (something like runners).
Proposal: Extend Group-Level config page
.gitlab-ci.yml syntax
.gitlab-ci.yml syntax stays intact:
job:
environment: production
resource_group: production
Resource Selecting logic
- If a managed resource group
production
exists at project-level, a job attempts to allocate a resource from the project-level resource group. [OUT OF SCOPE] - If a managed resource group
production
exists at group-level, a job attempts to allocate a resource from the group-level resource group. [New] - If a managed resource group
production
exists at instance-level, a job attempts to allocate a resource from the instance-level resource group. [OUT OF SCOPE] - If an unmanaged resource group
production
exists at project-level, a job attempts to allocate a resource from the project-level resource group. [Same] - If
production
does not exist anywhere, a job creates a new unmanaged resource group in the project-level. [Same]
How to create a group-level resource group
- Visit "Group > Settings > CI/CD > Resource Group"
- Click "New resource group"
- Fill "name" e.g. "iOS"
- Fill "resource count" e.g. "2" [OUT OF SCOPE]
- Click "Apply"
NOTES:
- Only maintainer or above can access the page.
- Update/Delete action is OUT OF SCOPE due to a technical difficulty. (We don't even support it at project-level)
How to see the current resource assignment status
- Visit "Group > Settings > CI/CD > Resource Group"
- The page shows a table of existing resource group:
- name: iOS
- name: Android
How this table will be extended [OUT OF SCOPE]:
- name: iOS
- status: free/occupied
build_project: gitlab-org/gitlab
build_id: #11111
build_status: Running
- status: free/occupied # If `resource count` is `2`, it shows multiple rows under the resource group.
build_project: gitlab-org/gitaly
build_id: #22222
build_status: Running
- name: Android
- status: free/occupied
build_project: gitlab-org/workhorse
build_id: #33333
build_status: Running
NOTES:
- This table can be re-used in project-level config page. [OUT OF SCOPE]
Key points
- This follows the vision of this feature.
- We need to introduce UI (Configuration or even dedicated page) for this feature sooner or later.
- This feature will never be complete in gitlab-ci.yml level.
- It's secure, because it performs a permission check on the resource creation.
Side work
Regardless of the approach on this issue, we need to finish the following work:
Database table rearchitecting
Given the previous iteration was strictly sticking with MVC, the current architecture cannot simply be extended for the group-level. We need the following rearchitecting:
ci_resource_groups_projects ( # New table
project_id # FK
resource_group_id # FK
)
ci_resource_groups_groups ( # New table
group_id # FK
resource_group_id # FK
)
ci_resource_groups ( # Existing table
# project_id bigint NOT NULL, # To be removed, in favor of `ci_resource_groups_projects`
hierarchy_type int # enum. One of `project`, `group`, or `instance`.
key character varying(255) NOT NULL
)
We need to migrate ci_resource_groups.project_id
to the new table, or support both for the transition period.
Permissions and Security
Documentation
Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.