Use CTE optimization fence for loading projects in dashboard
Certain users experienced Error 500s when loading projects from the dashboard because the PostgreSQL query planner attempted to scan all projects rather than just the user's authorized projects. This would cause the query to hit a statement timeout.
To fix this, we add support for a CTE optimization fence to load
authorized projects first, which can be optionally used by
ProjectsFinder
via the use_cte
parameter. To be safe, we only enable
it for the finder call that loads the list of projects behind the
use_cte_for_projects_finder
feature flag.
Related to #198440 (closed)
Query plans
Before
SELECT
"projects".*
FROM
"projects"
INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id"
WHERE
"project_authorizations"."user_id" = 1614863
AND "projects"."archived" = FALSE
ORDER BY
"projects"."created_at" DESC
LIMIT
20 OFFSET 0
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1.12..9975.94 rows=20 width=774) (actual time=10.995..20002.791 rows=20 loops=1)
-> Nested Loop (cost=1.12..8468118.51 rows=16979 width=774) (actual time=10.995..20002.785 rows=20 loops=1)
-> Index Scan Backward using index_projects_api_created_at_id_desc on projects (cost=0.56..975185.36 rows=11558649 width=774) (actual time=0.032..12793.268 rows=3779934 loops=1)
Filter: (NOT archived)
Rows Removed by Filter: 52269
-> Index Only Scan using index_project_authorizations_on_user_id_project_id_access_level on project_authorizations (cost=0.56..0.64 rows=1 width=4) (actual time=0.002..0.002 rows=0 loops=3779934)
Index Cond: ((user_id = 1614863) AND (project_id = projects.id))
Heap Fetches: 3
Planning time: 5.621 ms
Execution time: 20002.920 ms
(10 rows)
After
WITH authorized_projects AS (
SELECT
"projects".*
FROM
"projects"
INNER JOIN "project_authorizations" ON "projects"."id" = "project_authorizations"."project_id"
WHERE
"project_authorizations"."user_id" = 1614863
)
SELECT
*
FROM
authorized_projects
WHERE
archived = 'false'
ORDER BY
created_at
LIMIT
20 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=57619.49..57619.54 rows=20 width=4316) (actual time=136.124..136.130 rows=20 loops=1)
CTE authorized_projects
-> Nested Loop (cost=1.00..57043.39 rows=17298 width=774) (actual time=0.047..103.892 rows=17861 loops=1)
-> Index Only Scan using index_project_authorizations_on_user_id_project_id_access_level on project_authorizations (cost=0.56..1119.56 rows=17298 width=4) (actual time=0.028..5.546 rows=17861 loops=1)
Index Cond: (user_id = 1614863)
Heap Fetches: 1123
-> Index Scan using projects_pkey on projects (cost=0.43..3.22 rows=1 width=774) (actual time=0.005..0.005 rows=1 loops=17861)
Index Cond: (id = project_authorizations.project_id)
-> Sort (cost=576.11..597.73 rows=8649 width=4316) (actual time=136.122..136.125 rows=20 loops=1)
Sort Key: authorized_projects.created_at
Sort Method: top-N heapsort Memory: 35kB
-> CTE Scan on authorized_projects (cost=0.00..345.96 rows=8649 width=4316) (actual time=0.053..133.490 rows=17861 loops=1)
Filter: (NOT archived)
Planning time: 0.961 ms
Execution time: 138.357 ms
(15 rows)
Edited by Mayra Cabrera