Backend: API allowing projects to add groups in their CI_JOB_TOKEN allowlist

Create an API option to enable projects to add groups to their CI_JOB_TOKEN allow list. Focusing on the API over UI option as it is likely users will want to script/automate this work.

Additional details:

• This will be offered via GraphQL
• The current 200 project limit will be a combination of project + groups (in other words, the total number of projects in a group will not count toward the 200 limit - only the group itself in the list applies as 1).
• Groups are dynamic and on the backend, we will need to track as appropriate.
• Per feedback: We probably need some method to automatically update reference links when things move around, especially in terms of group accesses. The concern here is the current implementation is static and it for enterprise users, it is highly likely that subgroups or project paths will change with re-organization. Having a way to detect and as an MVC provide error handling will at least notify users that a path is now broken.
• In preparation for UI work, we should also capture the total number of projects in the group.

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

added to epic &11337

changed the description

added CI job token Category:Secrets Management featureaddition grouppipeline security typefeature labels

added devopsverify sectionci labels

changed milestone to %16.9

added workflowrefinement label

changed the description

added workflowplanning breakdown label and removed workflowrefinement label

added needs weight label

@shampton I updated this per our discussion - please add details you think are critical for implementation. Thanks!

changed the description

changed the description

added Deliverable label

mentioned in issue gitlab-com/Product#13036 (closed)

assigned to @dbiryukov

Setting health status to on track as the milestone has just begun.

Issue participants are welcome to override this by setting the health status to another value.

changed health status to on track

I am very much interested in this feature, because we will be hitting 200 project limit for some quite general projects very soon, when this gets enforced. It is no way to bypass the 200 project limit (to increase it to 1000)? The reason for this is because there are general "configuration" and "process" submodules, which are referenced by every project we have.

This limitation of 200 project limit (and the limitation of the CI_JOB_TOKEN) creates a significant impact on our extensive usage of git submodules.

Technical proposition.

We have to implement a group table in the allowlist.

Allow list implementation:

1. Similar to Ci::JobToken::ProjectScopeLink, should be introduced Ci::JobToken::GroupScopeLink model/table, source_project_id, target_group_id
2. Extend Ci::JobToken::Allowlist to support with new methods add_group and includes_group?(target_project)
3. Inject to Ci::JobToken::Scope accessible? method, a condition `group_accessible?(accessed_project)
4. Implement group_accessible with the scope Project.for_group_and_its_subgroups(target_group), with target_group iterated from Ci::JobToken::GroupScopeLink of accessible project allow list.

The primary performance bottleneck lies in the iteration of group_accessible through all the groups listed in the allowlist and mapping all the projects using Project.for_group_and_its_subgroups(target_group). An alternative consideration is to perform a direct sql/activerecord request for each group in the allowlist, checking if the project belongs to that specific group.

API/GraphQL

1. Introduce a mutation similar to Mutations::Ci::JobTokenScope::AddProject -> Mutations::Ci::JobTokenScope::AddGroup
2. Mutations::Ci::JobTokenScope::RemoveGroup

Per feedback: We probably need some method to automatically update reference links when things move around, especially in terms of group accesses. The concern here is the current implementation is static and it for enterprise users, it is highly likely that subgroups or project paths will change with re-organization. Having a way to detect and as an MVC provide error handling will at least notify users that a path is now broken.

For the initial MVP, I propose omitting this step. In the event of changes to subgroups or project paths, it is challenging to promptly notify users of potential broken paths. The current implementation automatically removes a project from the allowlist if it is deleted. I don't foresee scenarios where paths could break, except in cases of deleting a specific group. The lists of projects and subgroups will dynamically expose themselves based on the root group included in the allowlist.

I anticipate the backend portion to have a weight of approximately 3.

cc: @shampton @jocelynjane

@dbiryukov - I agree, the solution for the reference link problem should be addressed in a subsequent iteration. Our first goal right now is to enable groups v. projects. What would it take though to make sure we have robust enough error messaging to notify the user a path is no longer valid (assuming it was previously)?

Great, thanks for the subsequent issue! I will estimate it, but implementation would depend more on the design, when, and how we want to know if the path got broken.

@dbiryukov I had a thought for how we could potentially check that the accessing project is in the allowed group.

If we can get the full path of the project (e.g. gitlab-org/gitlab for this project) and the full path of the allowed group, then we could theoretically do a string comparison. For example, if I added gitlab-org to my allowed list and gitlab-org/gitlab is trying to access my project then we could check that gitlab-org is at the beginning of gitlab-org/gitlab.

That said, I'm not entirely sure if that is secure or feasible. So I just wanted to put that out there to see what your thoughts were.

@shampton Sound like a great idea! Will elaborate on it more.