Skip to content

Add index to projects on namespace_id and repository_size_limit

Robert May requested to merge add-repository-size-limit-index into master

What does this MR do and why?

Add index to projects on namespace_id and repository_size_limit. This will improve a slow query present on pretty much every page load on GitLab.com, caused by a query in ee/app/views/shared/_namespace_storage_limit_alert.html.haml

Changelog: performance

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Screenshot_2023-07-24_at_16.37.04

Screenshot_2023-07-24_at_16.19.09

Previous query plan on .com

 Limit  (cost=22636.06..22636.07 rows=1 width=32) (actual time=14833.165..14833.176 rows=1 loops=1)
   Buffers: shared hit=12069 read=10209 dirtied=401
   I/O Timings: read=14578.741 write=0.000
   ->  Aggregate  (cost=22636.06..22636.07 rows=1 width=32) (actual time=14833.163..14833.171 rows=1 loops=1)
         Buffers: shared hit=12069 read=10209 dirtied=401
         I/O Timings: read=14578.741 write=0.000
         ->  Nested Loop  (cost=858.15..22611.45 rows=3282 width=24) (actual time=4720.410..14833.130 rows=1 loops=1)
               Buffers: shared hit=12069 read=10209 dirtied=401
               I/O Timings: read=14578.741 write=0.000
               ->  Nested Loop  (cost=857.59..16310.31 rows=10039 width=12) (actual time=838.306..7059.434 rows=3143 loops=1)
                     Buffers: shared hit=2091 read=4442 dirtied=176
                     I/O Timings: read=6946.211 write=0.000
                     ->  HashAggregate  (cost=857.02..861.10 rows=408 width=28) (actual time=826.554..831.513 rows=674 loops=1)
                           Group Key: namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)]
                           Buffers: shared hit=1 read=654 dirtied=18
                           I/O Timings: read=806.766 write=0.000
                           ->  Bitmap Heap Scan on public.namespaces  (cost=217.88..856.00 rows=408 width=28) (actual time=97.513..824.636 rows=674 loops=1)
                                 Buffers: shared hit=1 read=654 dirtied=18
                                 I/O Timings: read=806.766 write=0.000
                                 ->  Bitmap Index Scan using index_namespaces_on_traversal_ids_for_groups  (cost=0.00..217.78 rows=408 width=0) (actual time=94.752..94.754 rows=676 loops=1)
                                       Index Cond: (namespaces.traversal_ids @> '{9970}'::integer[])
                                       Buffers: shared hit=1 read=142
                                       I/O Timings: read=89.981 write=0.000
                     ->  Index Scan using index_projects_on_namespace_id_and_id on public.projects  (cost=0.56..37.62 rows=25 width=16) (actual time=2.268..9.229 rows=5 loops=674)
                           Index Cond: (projects.namespace_id = (namespaces.traversal_ids)[array_length(namespaces.traversal_ids, 1)])
                           Filter: ((projects.repository_size_limit <> 0) OR (projects.repository_size_limit IS NULL))
                           Rows Removed by Filter: 0
                           Buffers: shared hit=2090 read=3788 dirtied=158
                           I/O Timings: read=6139.445 write=0.000
               ->  Index Scan using index_project_statistics_on_project_id on public.project_statistics  (cost=0.56..0.62 rows=1 width=20) (actual time=2.469..2.469 rows=0 loops=3143)
                     Index Cond: (project_statistics.project_id = projects.id)
                     Filter: ((project_statistics.repository_size + project_statistics.lfs_objects_size) > COALESCE(projects.repository_size_limit, '10737418240'::bigint))
                     Rows Removed by Filter: 1
                     Buffers: shared hit=9978 read=5767 dirtied=225
                     I/O Timings: read=7632.531 write=0.000

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Robert May

Merge request reports