Fix projects new page performance

What does this MR do?

Related issue: #33188 (closed)

Fix project new page load performance, when group templates are visible.

You can see how slow the current page is by navigating to https://gitlab.com/projects/new

Same experience when "Create from Template" is clicked and "Group" tab is selected.

Feature Flag

Additional test cases have been added to ensure that the optimized version of the query works the same way as the original, however to be safe, the change is behind a feature flag.

Feature.enable(:optimized_groups_with_templates_finder)

Cleanup issue: #35733 (closed)

Queries

Count, without group

New, optimized:

SELECT COUNT(*)
FROM "projects"
WHERE "projects"."namespace_id" IN
    (WITH RECURSIVE "base_and_descendants" AS (
                                                 (SELECT "namespaces".*
                                                  FROM "namespaces"
                                                  INNER JOIN "members" ON "namespaces"."id" = "members"."source_id"
                                                  WHERE "members"."type" IN ('GroupMember')
                                                    AND "members"."source_type" = 'Namespace'
                                                    AND "namespaces"."type" IN ('Group')
                                                    AND "members"."user_id" = 4156052
                                                    AND "members"."requested_at" IS NULL
                                                    AND (members.access_level >= 30))
                                               UNION
                                                 (SELECT "namespaces".*
                                                  FROM "namespaces",
                                                       "base_and_descendants"
                                                  WHERE "namespaces"."type" IN ('Group')
                                                    AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT DISTINCT "namespaces"."id"
     FROM "base_and_descendants" AS "namespaces"
     INNER JOIN "projects" ON "projects"."namespace_id" = "namespaces"."id"
     WHERE "namespaces"."id" IN
         (WITH RECURSIVE "base_and_descendants" AS (
                                                      (WITH RECURSIVE "base_and_ancestors" AS (
                                                                                                 (SELECT "namespaces".*
                                                                                                  FROM "namespaces"
                                                                                                  WHERE "namespaces"."type" IN ('Group')
                                                                                                    AND (custom_project_templates_group_id IS NOT NULL))
                                                                                               UNION
                                                                                                 (SELECT "namespaces".*
                                                                                                  FROM "namespaces",
                                                                                                       "base_and_ancestors"
                                                                                                  WHERE "namespaces"."type" IN ('Group')
                                                                                                    AND "namespaces"."id" = "base_and_ancestors"."parent_id")) SELECT "namespaces".*
                                                       FROM "base_and_ancestors" AS "namespaces"
                                                       WHERE (EXISTS
                                                                (SELECT 1
                                                                 FROM "plans"
                                                                 INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id"
                                                                 WHERE "plans"."name" IN ('silver',
                                                                                          'gold')
                                                                   AND (gitlab_subscriptions.namespace_id = namespaces.id))))
                                                    UNION
                                                      (SELECT "namespaces".*
                                                       FROM "namespaces",
                                                            "base_and_descendants"
                                                       WHERE "namespaces"."type" IN ('Group')
                                                         AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT "custom_project_templates_group_id"
          FROM "base_and_descendants" AS "namespaces"
          WHERE (custom_project_templates_group_id IS NOT NULL)))

Plan

Current, slow:

SELECT COUNT(*)
FROM "projects"
WHERE "projects"."namespace_id" IN
    (WITH RECURSIVE "base_and_descendants" AS (
                                                 (SELECT "namespaces".*
                                                  FROM "namespaces"
                                                  INNER JOIN "members" ON "namespaces"."id" = "members"."source_id"
                                                  WHERE "members"."type" IN ('GroupMember')
                                                    AND "members"."source_type" = 'Namespace'
                                                    AND "namespaces"."type" IN ('Group')
                                                    AND "members"."user_id" = 2535118
                                                    AND "members"."requested_at" IS NULL
                                                    AND (members.access_level >= 30))
                                               UNION
                                                 (SELECT "namespaces".*
                                                  FROM "namespaces",
                                                       "base_and_descendants"
                                                  WHERE "namespaces"."type" IN ('Group')
                                                    AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT DISTINCT "namespaces"."id"
     FROM "base_and_descendants" AS "namespaces"
     WHERE "namespaces"."id" IN
         (WITH RECURSIVE "base_and_descendants" AS (
                                                      (SELECT "namespaces".*
                                                       FROM "namespaces"
                                                       WHERE "namespaces"."type" IN ('Group')
                                                         AND (EXISTS
                                                                (SELECT 1
                                                                 FROM "plans"
                                                                 INNER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."hosted_plan_id" = "plans"."id"
                                                                 WHERE "plans"."name" IN ('silver',
                                                                                          'gold')
                                                                   AND (gitlab_subscriptions.namespace_id = namespaces.id))))
                                                    UNION
                                                      (SELECT "namespaces".*
                                                       FROM "namespaces",
                                                            "base_and_descendants"
                                                       WHERE "namespaces"."type" IN ('Group')
                                                         AND "namespaces"."parent_id" = "base_and_descendants"."id")) SELECT DISTINCT "custom_project_templates_group_id"
          FROM "base_and_descendants" AS "namespaces"
          INNER JOIN projects ON projects.namespace_id = namespaces.custom_project_templates_group_id))

Plan

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Performance and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team