Let an Owner choose the Duo group of a Slack workspace

Jannik Lehmannrequested to merge
633566-duo-group-services into master

What does this MR do and why?

MR 3 of the split for #633566 (GitLab Duo in Slack: one Duo group per Slack workspace). It adds the rule for who may claim a Slack workspace for a top-level group, and the services that turn GitLab Duo in Slack on and off. It is the first MR that writes rows to slack_workspaces (table from !263649 (merged)).

  • Integrations::SlackWorkspace::DuoGroupEligibility: a top-level group may claim a workspace when the app is installed somewhere in its hierarchy (Owner decides) or for the instance (administrator decides, self-managed only). Checked with two EXISTS queries, so a mention never loads every install of a workspace.
  • EnableDuoGroupService and ReleaseDuoGroupService. Enable takes over a claim that no longer holds and refuses a valid claim of another group; a lost race for the same group counts as success. Release is the system path for a claim the group can no longer hold. DisableDuoGroupService comes with its caller, the group settings page (MR 5).
  • An instance-installed workspace (self-managed) is always refused with :managed_by_admin. Until the Admin area (MR 7) lands, no group can claim such a workspace; that is intended, the administrator decides where GitLab Duo runs for it.
  • New update_duo_in_slack ability for Owners of top-level groups, listed in group:archived so archived groups and groups scheduled for deletion cannot use it.
  • Audit events duo_in_slack_enabled and duo_in_slack_disabled.
  • Installing the app on a top-level group claims the workspace, so an Owner needs no second step. A reinstall does not claim again. The claim never fails the install, because the OAuth code is single-use by then.
  • Two partial indexes on slack_integrations, so the eligibility check stays cheap for a hierarchy without an install of the workspace (see Database).
  • Feature flag slack_duo_group (wip, default off). Actor: the top-level group, so it can be turned on for one group first.

Everything is inert while the flag is off. The mention path (MR 4), the group page (MR 5/6), the Admin area (MR 7) and the install page (MR 8) are the callers; they follow once this is in.

Most of the code comes from the reference implementation !262856 (closed). Changes against it:

  • Flag actor is the top-level group, not :instance.
  • The ~is_root_namespace prevent is folded into the existing block in EE::GroupPolicy; scheduled-for-deletion is covered by the group:archived permission group.
  • Only code with a caller in this MR is included. The reference implementation's helpers for the pages and the mention path (root_namespace_ids_by_team, team_names, team_ids_with_scope, with_duo_namespace_details, released_by_removing?, the preloaded arguments of DuoGroupEligibility) come with MR 4/5/7, so no keela_excluded.yml entries are needed here.
  • Enable rescues ActiveRecord::RecordInvalid as well as RecordNotUnique: when another group claims the workspace between the lookup and the insert, the team_id uniqueness validation raises first; the index only raises when both inserts interleave.

References

Screenshots or screen recordings

No UI in this MR.

How to set up and validate locally

The services have no UI yet. Use the Rails console.

  1. Enable the flag for a top-level group you own: 
    group = Group.find_by_full_path('your-top-level-group')
Feature.enable(:slack_duo_group, group)
  2. Give the group a fake Slack install (or install the real app on the group, which claims the workspace on its own): 
    integration = Integrations::GitlabSlackApplication.create!(group: group, active: true)
SlackIntegration.create!(integration: integration, group: group, team_id: 'T_LOCAL', team_name: 'Local', alias: group.full_path, user_id: 'U1', bot_user_id: 'UBOT', bot_access_token: 'xoxb-local')
  3. Turn GitLab Duo on, then release the claim the way the system does once the group can no longer hold it: 
    owner = group.owners.first
Integrations::SlackWorkspace::EnableDuoGroupService.new(group, current_user: owner, team_id: 'T_LOCAL').execute
workspace = Integrations::SlackWorkspace::Workspace.for_team('T_LOCAL')
workspace.duo_namespace == group # => true
group.namespace_settings.update!(archived: true)
Integrations::SlackWorkspace::ReleaseDuoGroupService.for(workspace).execute # => released: true
  4. Check the audit events for the group under Secure > Audit events: duo_in_slack_enabled and duo_in_slack_disabled by (System).
  5. Try it as a Maintainer (:forbidden), on a subgroup (:not_top_level_group), or with a second top-level group that has the same team_id installed (:claimed_by_another_group).

For every state at once, the seed script in #633566 (comment 4012898135) builds a test world.

Database

Below are the migration and all statements this MR adds, captured from Rails and explained on postgres.ai (gitlab-production-main) for the gitlab-org hierarchy (namespace_id = 9970, about 1,640 groups and 9,280 projects). The Slack workspace is written as <team_id> here. The plans for 1 and 2 use the literal value, as Rails sends it. The others pick it with a scalar subquery (InitPlan 1); they are single-row lookups or writes, so their plans do not depend on how many installs a workspace has.

slack_workspaces is empty in production and did not exist on the clone snapshot yet, so for 4–9 the table and its indexes were created from db/structure.sql and seeded with 5,661 rows (one per top-level group) before explaining. The clone was reset afterwards.

Migration

20261101192034_add_team_indexes_with_bot_to_slack_integrations.rb adds two partial indexes for 1 and 2 (requested in !265962 (comment 4123959142)):

CREATE INDEX CONCURRENTLY "index_slack_integrations_on_team_id_and_group_id_with_bot" ON "slack_integrations" ("team_id", "group_id") WHERE bot_user_id IS NOT NULL;
CREATE INDEX CONCURRENTLY "index_slack_integrations_on_team_id_and_project_id_with_bot" ON "slack_integrations" ("team_id", "project_id") WHERE bot_user_id IS NOT NULL;
  • db:gitlabcom-database-testing: 9.1 s, +22.08 MiB (!265962 (comment 4136245735)). slack_integrations has about 513,000 rows and is not on the large tables list.
  • Rollback: down removes both indexes. Tested locally with db:migrate:up:main and db:migrate:down:main.

1. DuoGroupEligibility#installed_in_hierarchy?, installs on groups

SlackIntegration.with_bot.by_team(team_id).for_groups_in(group).exists?

SELECT 1 AS one FROM "slack_integrations"
WHERE "slack_integrations"."bot_user_id" IS NOT NULL
  AND "slack_integrations"."team_id" = '<team_id>'
  AND "slack_integrations"."group_id" IN (
    SELECT "namespaces"."id" FROM UNNEST(COALESCE(
      (SELECT ids FROM (SELECT "namespace_descendants"."self_and_descendant_group_ids" AS ids FROM "namespace_descendants" WHERE "namespace_descendants"."outdated_at" IS NULL AND "namespace_descendants"."namespace_id" = <group_id>) cached_query),
      (SELECT ids FROM (SELECT ARRAY_AGG("namespaces"."id") AS ids FROM (SELECT namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] AS id FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND (traversal_ids @> ('{<group_id>}'))) namespaces) consistent_query)
    )) AS namespaces(id))
LIMIT 1

Plans with index_slack_integrations_on_team_id_and_group_id_with_bot:

The planner estimates the UNNEST at 10 rows and probes the index once per group of the hierarchy, 1,646 times when nothing matches. Every probe is index-only (Heap Fetches: 0). Without the new index the same probes go through index_slack_integrations_on_group_id and read the table row to check team_id and bot_user_id: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/75941/commands/182143 (32 ms, 152 buffers) and https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/75941/commands/182144 (160 ms, 5,003 buffers, 222 of them read).

2. DuoGroupEligibility#installed_in_hierarchy?, installs on projects

SlackIntegration.with_bot.by_team(team_id).for_projects_in(group).exists?

SELECT 1 AS one FROM "slack_integrations"
WHERE "slack_integrations"."bot_user_id" IS NOT NULL
  AND "slack_integrations"."team_id" = '<team_id>'
  AND "slack_integrations"."project_id" IN (
    SELECT "projects"."id" FROM UNNEST(COALESCE(
      (SELECT ids FROM (SELECT "namespace_descendants"."all_project_ids" AS ids FROM "namespace_descendants" WHERE "namespace_descendants"."outdated_at" IS NULL AND "namespace_descendants"."namespace_id" = <group_id>) cached_query),
      (SELECT ids FROM (SELECT ARRAY_AGG("projects"."id") AS ids FROM (SELECT "projects"."id" FROM "projects" WHERE "projects"."namespace_id" IN (
        SELECT "namespaces"."id" FROM UNNEST(COALESCE(
          (SELECT ids FROM (SELECT "namespace_descendants"."self_and_descendant_group_ids" AS ids FROM "namespace_descendants" WHERE "namespace_descendants"."outdated_at" IS NULL AND "namespace_descendants"."namespace_id" = <group_id>) cached_query),
          (SELECT ids FROM (SELECT ARRAY_AGG("namespaces"."id") AS ids FROM (SELECT namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] AS id FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND (traversal_ids @> ('{<group_id>}'))) namespaces) consistent_query)
        )) AS namespaces(id))) projects) consistent_query)
    )) AS projects(id))
LIMIT 1

Plans with index_slack_integrations_on_team_id_and_project_id_with_bot:

Same shape as 1 with the all_project_ids array: 9,285 index-only probes when nothing matches. Without the new index they go through index_slack_integrations_on_project_id and read a table row each: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/75941/commands/182145 (27 ms, 94 buffers) and https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/75941/commands/182146 (1,154 ms, 28,329 buffers, 1,619 of them read). The statement on the clone is this query with ActiveRecord's identifier quoting and aliases shortened; the CLI rejects arguments over about 1 KB.

3. DuoGroupEligibility.instance_install?

SlackIntegration.with_bot.by_team(team_id).instance_level_in(organization).exists?

SELECT 1 AS one FROM "slack_integrations"
WHERE "slack_integrations"."bot_user_id" IS NOT NULL
  AND "slack_integrations"."team_id" = '<team_id>'
  AND "slack_integrations"."group_id" IS NULL
  AND "slack_integrations"."project_id" IS NULL
  AND "slack_integrations"."organization_id" = <organization_id>
LIMIT 1

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/74330/commands/178268 (execution: 0.08 ms, 6 buffers, 0 rows)

GitLab.com has no instance-level installs and DuoGroupEligibility.instance_installs? is false there, so this runs on self-managed only. index_slack_integrations_on_organization_id narrows it to the organization's instance-level rows.

4. Workspace.for_team (Enable)

SELECT "slack_workspaces".* FROM "slack_workspaces" WHERE "slack_workspaces"."team_id" = '<team_id>' LIMIT 1

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/74330/commands/178285 (execution: 0.07 ms, 3 buffers, 1 row), on the unique index_slack_workspaces_on_team_id.

5. team_id uniqueness validation before the insert (Enable)

SELECT 1 AS one FROM "slack_workspaces" WHERE "slack_workspaces"."team_id" = '<team_id>' LIMIT 1

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/74330/commands/178286 (execution: 0.05 ms, 2 buffers), index-only scan on the same index.

6. Workspace.create! (Enable)

INSERT INTO "slack_workspaces" ("duo_namespace_id", "created_at", "updated_at", "team_id") VALUES (<group_id>, '<now>', '<now>', '<team_id>') RETURNING "id"

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/74330/commands/178287 (execution: 3.3 ms, 104 buffers). One row, two indexes and the foreign key on duo_namespace_id to maintain.

7. Audit target_details (all three services)

SlackIntegration.by_team(team_id).pick(:team_name)

SELECT "slack_integrations"."team_name" FROM "slack_integrations" WHERE "slack_integrations"."team_id" = '<team_id>' LIMIT 1

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/74330/commands/178288 (execution: 0.3 ms, 49 buffers, 1 row), on index_slack_integrations_on_team_id_alias_project_id.

8. ReleaseDuoGroupService (Workspace.with_duo_namespace(group).for_team(team_id), then destroy!)

SELECT "slack_workspaces".* FROM "slack_workspaces" WHERE "slack_workspaces"."duo_namespace_id" = <group_id> AND "slack_workspaces"."team_id" = '<team_id>' LIMIT 1

Then the same DELETE … WHERE "id" = <id> as 9. It was a delete_all before; the row is destroyed through the model so that the Cells claim on team_id (#633611) is released by its callback. At most one row, because team_id is unique.

Plan for the lookup: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/74330/commands/178289 shows the same index scan on the unique team_id index that the former DELETE used (0.14 ms, 5 buffers); the SELECT reads the same row.

9. ReleaseDuoGroupService (workspace.destroy!)

DELETE FROM "slack_workspaces" WHERE "slack_workspaces"."id" = <id>

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/74330/commands/178290 (execution: 0.18 ms, 8 buffers, 1 row deleted), by primary key.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jannik Lehmann

Merge request reports