Skip to content

Use CTE optimization fence for loading projects in dashboard

Stan Hu requested to merge sh-use-cte-for-projects-finder into master

Certain users experienced Error 500s when loading projects from the dashboard because the PostgreSQL query planner attempted to scan all projects rather than just the user's authorized projects. This would cause the query to hit a statement timeout.

To fix this, we add support for a CTE optimization fence to load authorized projects first, which can be optionally used by ProjectsFinder via the use_cte parameter. To be safe, we only enable it for the finder call that loads the list of projects behind the use_cte_for_projects_finder feature flag.

Related to #198440 (closed)

Query plans

Before

SELECT
  "projects".*
FROM
  "projects"
  INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id"
WHERE
  "project_authorizations"."user_id" = 1614863
  AND "projects"."archived" = FALSE
ORDER BY
  "projects"."created_at" DESC
LIMIT
  20 OFFSET 0
                                                                                                  QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1.12..9975.94 rows=20 width=774) (actual time=10.995..20002.791 rows=20 loops=1)
   ->  Nested Loop  (cost=1.12..8468118.51 rows=16979 width=774) (actual time=10.995..20002.785 rows=20 loops=1)
         ->  Index Scan Backward using index_projects_api_created_at_id_desc on projects  (cost=0.56..975185.36 rows=11558649 width=774) (actual time=0.032..12793.268 rows=3779934 loops=1)
               Filter: (NOT archived)
               Rows Removed by Filter: 52269
         ->  Index Only Scan using index_project_authorizations_on_user_id_project_id_access_level on project_authorizations  (cost=0.56..0.64 rows=1 width=4) (actual time=0.002..0.002 rows=0 loops=3779934)
               Index Cond: ((user_id = 1614863) AND (project_id = projects.id))
               Heap Fetches: 3
 Planning time: 5.621 ms
 Execution time: 20002.920 ms
(10 rows)

After

WITH authorized_projects AS (
  SELECT
    "projects".*
  FROM
    "projects"
    INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id"
  WHERE
    "project_authorizations"."user_id" = 1614863
)
SELECT
  *
FROM
  authorized_projects
WHERE
  archived = 'false'
ORDER BY
  created_at
LIMIT
  20 OFFSET 0;
                                                                                                      QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=57619.49..57619.54 rows=20 width=4316) (actual time=136.124..136.130 rows=20 loops=1)
   CTE authorized_projects
     ->  Nested Loop  (cost=1.00..57043.39 rows=17298 width=774) (actual time=0.047..103.892 rows=17861 loops=1)
           ->  Index Only Scan using index_project_authorizations_on_user_id_project_id_access_level on project_authorizations  (cost=0.56..1119.56 rows=17298 width=4) (actual time=0.028..5.546 rows=17861 loops=1)
                 Index Cond: (user_id = 1614863)
                 Heap Fetches: 1123
           ->  Index Scan using projects_pkey on projects  (cost=0.43..3.22 rows=1 width=774) (actual time=0.005..0.005 rows=1 loops=17861)
                 Index Cond: (id = project_authorizations.project_id)
   ->  Sort  (cost=576.11..597.73 rows=8649 width=4316) (actual time=136.122..136.125 rows=20 loops=1)
         Sort Key: authorized_projects.created_at
         Sort Method: top-N heapsort  Memory: 35kB
         ->  CTE Scan on authorized_projects  (cost=0.00..345.96 rows=8649 width=4316) (actual time=0.053..133.490 rows=17861 loops=1)
               Filter: (NOT archived)
 Planning time: 0.961 ms
 Execution time: 138.357 ms
(15 rows)
Edited by Mayra Cabrera

Merge request reports