Create an API option to enable projects to add groups to their CI_JOB_TOKEN allow list. Focusing on the API over UI option as it is likely users will want to script/automate this work.
Additional details:
This will be offered via GraphQL
The current 200 project limit will be a combination of project + groups (in other words, the total number of projects in a group will not count toward the 200 limit - only the group itself in the list applies as 1).
Groups are dynamic and on the backend, we will need to track as appropriate.
Per feedback: We probably need some method to automatically update reference links when things move around, especially in terms of group accesses. The concern here is the current implementation is static and it for enterprise users, it is highly likely that subgroups or project paths will change with re-organization. Having a way to detect and as an MVC provide error handling will at least notify users that a path is now broken.
In preparation for UI work, we should also capture the total number of projects in the group.
This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
I am very much interested in this feature, because we will be hitting 200 project limit for some quite general projects very soon, when this gets enforced. It is no way to bypass the 200 project limit (to increase it to 1000)? The reason for this is because there are general "configuration" and "process" submodules, which are referenced by every project we have.
This limitation of 200 project limit (and the limitation of the CI_JOB_TOKEN) creates a significant impact on our extensive usage of git submodules.
We have to implement a group table in the allowlist.
Allow list implementation:
Similar to Ci::JobToken::ProjectScopeLink, should be introduced Ci::JobToken::GroupScopeLink model/table, source_project_id, target_group_id
Extend Ci::JobToken::Allowlist to support with new methods add_group and includes_group?(target_project)
Inject to Ci::JobToken::Scopeaccessible? method, a condition `group_accessible?(accessed_project)
Implement group_accessible with the scope Project.for_group_and_its_subgroups(target_group), with target_group iterated from Ci::JobToken::GroupScopeLink of accessible project allow list.
The primary performance bottleneck lies in the iteration of group_accessible through all the groups listed in the allowlist and mapping all the projects using Project.for_group_and_its_subgroups(target_group). An alternative consideration is to perform a direct sql/activerecord request for each group in the allowlist, checking if the project belongs to that specific group.
API/GraphQL
Introduce a mutation similar to Mutations::Ci::JobTokenScope::AddProject -> Mutations::Ci::JobTokenScope::AddGroup
Mutations::Ci::JobTokenScope::RemoveGroup
Per feedback: We probably need some method to automatically update reference links when things move around, especially in terms of group accesses. The concern here is the current implementation is static and it for enterprise users, it is highly likely that subgroups or project paths will change with re-organization. Having a way to detect and as an MVC provide error handling will at least notify users that a path is now broken.
For the initial MVP, I propose omitting this step. In the event of changes to subgroups or project paths, it is challenging to promptly notify users of potential broken paths. The current implementation automatically removes a project from the allowlist if it is deleted. I don't foresee scenarios where paths could break, except in cases of deleting a specific group. The lists of projects and subgroups will dynamically expose themselves based on the root group included in the allowlist.
I anticipate the backend portion to have a weight of approximately 3.
@dbiryukov - I agree, the solution for the reference link problem should be addressed in a subsequent iteration. Our first goal right now is to enable groups v. projects. What would it take though to make sure we have robust enough error messaging to notify the user a path is no longer valid (assuming it was previously)?
Great, thanks for the subsequent issue! I will estimate it, but implementation would depend more on the design, when, and how we want to know if the path got broken.
@dbiryukov I had a thought for how we could potentially check that the accessing project is in the allowed group.
If we can get the full path of the project (e.g. gitlab-org/gitlab for this project) and the full path of the allowed group, then we could theoretically do a string comparison. For example, if I added gitlab-org to my allowed list and gitlab-org/gitlab is trying to access my project then we could check that gitlab-org is at the beginning of gitlab-org/gitlab.
That said, I'm not entirely sure if that is secure or feasible. 🤔 So I just wanted to put that out there to see what your thoughts were.
I'm dividing the implementation into four iterations to ease the review process.
Database design of group list table and model - job_token/group_scope_link
Add into app/models/ci/job_token/group_scope_link.rb and app/models/ci/job_token/allowlist.rb groups validation together with inbound lists, Feature flag group_scope_link_allowlist
1st iteration with database design and model definition got LGTM from database reviewers, waiting for backend review.
2nd iteration - validation implementation in progress.
Targeting to release for the review allowlist group validation and GraphQL API next week.