BE: Support mapped/unmapped agents for new authorization strategy for RD
Issue: #455804
What does this MR do and why?
- Update the query API to return unmapped/mapped cluster agent for a given namespace
- Implement specs to verify the changes
- Introduce a deterministic ordering for the cluster agents being returned i.e. by name
Query plans
Since the API supports distinct filter types that cannot be used at the same time, their query plans have been evaluated independently:
Fetching agents when filter == UNMAPPED
This is an admin API and its usage is expected to be very limited. The below snippet illustrates the SQL calls that are made when the core invocations are executed in IRB:
[10] pry(main)> existing_mapped_agents = RemoteDevelopment::RemoteDevelopmentNamespaceClusterAgentMapping.for_namespaces([24]).map(&:cluster_agent_id)
RemoteDevelopment::RemoteDevelopmentNamespaceClusterAgentMapping Load (1.6ms) SELECT "remote_development_namespace_cluster_agent_mappings".* FROM "remote_development_namespace_cluster_agent_mappings" WHERE "remote_development_namespace_cluster_agent_mappings"."namespace_id" = 24 /*application:console,db_config_name:main,console_hostname:Hunars-MacBook-Pro.local,console_username:hkhanna,line:(pry):10:in `map'*/
=> [2]
[11] pry(main)> Namespace.find_by(path: "gitlab-org").cluster_agents.id_not_in(existing_mapped_agents)
Namespace Load (0.9ms) SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."path" = 'gitlab-org' LIMIT 1 /*application:console,db_config_name:main,console_hostname:Hunars-MacBook-Pro.local,console_username:hkhanna,line:(pry):11:in `__pry__'*/
Clusters::Agent Load (1.9ms) SELECT "cluster_agents".* FROM "cluster_agents" WHERE "cluster_agents"."project_id" IN (SELECT "projects"."id" FROM "projects" WHERE "projects"."namespace_id" IN (SELECT namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] AS id FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND (traversal_ids @> ('{24}')))) AND "cluster_agents"."id" != 2 /*application:console,db_config_name:main,console_hostname:Hunars-MacBook-Pro.local,console_username:hkhanna,line:bin/rails:4:in `<main>'*/
=> [#<Clusters::Agent:0x000000016b3d21d0 id: 4, created_at: Thu, 22 Jun 2023 07:29:45.345253000 UTC +00:00, updated_at: Thu, 22 Jun 2023 07:29:45.345253000 UTC +00:00, project_id: 3, name: "something-else", created_by_user_id: 1, has_vulnerabilities: false>,
#<Clusters::Agent:0x000000016b3d2130 id: 3, created_at: Wed, 01 Mar 2023 07:08:58.521503000 UTC +00:00, updated_at: Wed, 01 Mar 2023 07:08:58.521503000 UTC +00:00, project_id: 3, name: "test-agent-2", created_by_user_id: 1, has_vulnerabilities: false>,
#<Clusters::Agent:0x000000016b3d2090 id: 10, created_at: Thu, 28 Mar 2024 03:46:46.897684000 UTC +00:00, updated_at: Thu, 28 Mar 2024 03:46:46.897684000 UTC +00:00, project_id: 3, name: "test-agent-bug", created_by_user_id: 1, has_vulnerabilities: false>]Raw SQL queries along with the query plan extracted from the above:
gitlabhq_development=# explain SELECT "remote_development_namespace_cluster_agent_mappings".* FROM "remote_development_namespace_cluster_agent_mappings" WHERE "remote_development_namespace_cluster_agent_mappings"."namespace_id" = 24;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using unique_namespace_cluster_agent_mappings_for_agent_association on remote_development_namespace_cluster_agent_mappings (cost=0.15..5.24 rows=5 width=48)
Index Cond: (namespace_id = 24)
(2 rows)
gitlabhq_development=# explain SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runne ", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."path" = 'gitlab-org' LIMIT 1;
QUERY PLAN
---------------------------------------------------------------------------------------------------
Limit (cost=0.27..2.29 rows=1 width=531)
-> Index Scan using index_namespaces_on_path on namespaces (cost=0.27..2.29 rows=1 width=531)
Index Cond: ((path)::text = 'gitlab-org'::text)
(3 rows)
gitlabhq_development=#Fetching agents when filter == DIRECTLY_MAPPED
gitlabhq_development=# explain select * from remote_development_namespace_cluster_agent_mappings where namespace_id in (24);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Index Scan using unique_namespace_cluster_agent_mappings_for_agent_association on remote_development_namespace_cluster_agent_mappings (cost=0.15..5.24 rows=5 width=48)
Index Cond: (namespace_id = 24)
(2 rows)
For the filtering of valid mappings:
gitlabhq_development=# explain select * from cluster_agents where id in (2);
QUERY PLAN
-------------------------------------------------------------------------------------------
Index Scan using cluster_agents_pkey on cluster_agents (cost=0.15..2.17 rows=1 width=73)
Index Cond: (id = 2)
(2 rows)
gitlabhq_development=# explain select * from projects where id in (3);
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------
Index Scan using idx_projects_on_repository_storage_last_repository_updated_at on projects (cost=0.14..2.16 rows=1 width=1443)
Index Cond: (id = 3)
(2 rows)
gitlabhq_development=# explain select * from namespaces where id in (26);
QUERY PLAN
------------------------------------------------------------------------------------
Index Scan using namespaces_pkey on namespaces (cost=0.14..2.16 rows=1 width=533)
Index Cond: (id = 26)
(2 rows)MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
| Before | After |
|---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Edited by Hunar Khanna