Allow projects to use agents from groups not directly related

What does this MR do and why?

Allows projects to use agents from groups outside of their immediate hierarchy.

Previously, for a project to use an agent via the CI tunnel, the agent's project had to belong directly to an ancestor group of the project using the agent.

With this change, any project can use any authorised agent, as long as they share a root ancestor.

Visual Example

This chart illustrates a hierarchy that was previously not possible, because the agent configuration project (Agent-Project) did not belong to a direct ancestor of the project wanting to use the agent (Feature-Project).

Now, since both projects share the Root-Group top-most group, this configuration is allowed.

graph TD;
  Root-Group-->Operators;
  Root-Group-->Developers;
  Developers-->Feature-Project;  
  Operators-->Agent-Project;

Database changes

There are two queries updated (one each for project and group level), and the actual change is the same for both. We update a projects subquery to match all projects in a hierarchy, instead of those belonging to a specific list of groups. Using traversal IDs (which are enabled on GitLab.com), this is a straightforward change from selecting where "namespaces"."id" IN (id1, id2, id3, ...)) to "namespaces"."traversal_ids" @> ('{rootId}')).

Initially this seems like it will be much slower, as an entire group hierarchy can have a far larger number of projects than a small list of groups. However, The examples below use the gitlab-org hierarchy, which has over 2000 projects, and we can see that this doesn't meaningfully impact the amount of data processed by the query, due to having a small number of relevant authorisation records to filter through.

Note that these examples only return a single row - this feature is not yet widely used, with only ~3000 and ~4000 total authorisation records on GitLab.com for projects and groups respectively. However, this is close to the expected typical usage, a single project is unlikely to be authorised to more than a handful of agents (1-3 would be the most common, >10 would be very rare).

Project level

• Before
• After

Group level

• Before
• After

How to set up and validate locally

1. Create a group/project hierarchy as shown above
2. Register an agent in Agent-Project, using the following configuration:

   ci_access:
     groups:
     - id: root-group/developers # Path to the group created in the previous step

3. Enable the feature flag using Feature.enable(:agent_authorization_include_descendants) from a Rails console
4. Run a CI pipeline in Feature-Project using the following .gitlab-ci.yml:

   stages:
     - test
   
   example-job:
     stage: test
     script:
       - echo "$KUBECONFIG"

5. Observe an entry matching the agent ID you created above.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

#346566 (closed)