Skip to content

Add index to improve counting of open issues

Heinrich Lee Yu requested to merge 363603-add-index-for-open-issues-count into master

What does this MR do and why?

Adds an index to improve the performance of counting open issues of a project. This query is triggered every time an issue is created or changes state.

This MR only prepares the index for weekend creation on GitLab.com: https://docs.gitlab.com/ee/development/adding_database_indexes.html#create-indexes-asynchronously

Query:

SELECT COUNT(*) AS count_all, "issues"."confidential" AS issues_confidential
FROM "issues"
WHERE ("issues"."state_id" IN (1)) AND "issues"."project_id" = 19923279
GROUP BY "issues"."confidential"

Note: The query right now in master includes an issue_type filter, but that will be removed by !89118 (merged)

Before:

 HashAggregate  (cost=51322.17..51322.19 rows=2 width=9) (actual time=249.499..249.501 rows=2 loops=1)
   Group Key: issues.confidential
   Buffers: shared hit=83043
   I/O Timings: read=0.000 write=0.000
   ->  Index Scan using index_issue_on_project_id_state_id_and_blocking_issues_count on public.issues  (cost=0.56..51142.76 rows=35883 width=1) (actual time=0.047..221.147 rows=91423 loops=1)
         Index Cond: ((issues.project_id = 19923279) AND (issues.state_id = 1))
         Buffers: shared hit=83043
         I/O Timings: read=0.000 write=0.000

Time: 253.770 ms  
  - planning: 4.148 ms  
  - execution: 249.622 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 83043 (~648.80 MiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0  

Sometimes the query is fast like above, but sometimes it times out as seen in the linked issue.

After:

 Aggregate  (cost=0.56..3454.36 rows=2 width=9) (actual time=0.243..55.344 rows=2 loops=1)
   Group Key: issues.confidential
   Buffers: shared hit=4931
   I/O Timings: read=0.000 write=0.000
   ->  Index Only Scan using idx_open_issues_on_project_id_and_confidential on public.issues  (cost=0.56..3274.93 rows=35882 width=1) (actual time=0.150..40.636 rows=91423 loops=1)
         Index Cond: (issues.project_id = 19923279)
         Heap Fetches: 4986
         Buffers: shared hit=4931
         I/O Timings: read=0.000 write=0.000

Time: 55.810 ms  
  - planning: 0.421 ms  
  - execution: 55.389 ms  
    - I/O read: 0.000 ms  
    - I/O write: 0.000 ms  
  
Shared buffers:  
  - hits: 4931 (~38.50 MiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0 
Migration output
main: == 20220602111923 PrepareIndexForOpenIssuesCount: migrating ===================
main: -- index_exists?(:issues, [:project_id, :confidential], {:where=>"state_id = 1", :name=>"idx_open_issues_on_project_id_and_confidential", :algorithm=>:concurrently})
main:    -> 0.0069s
main: -- add_index_options(:issues, [:project_id, :confidential], {:where=>"state_id = 1", :name=>"idx_open_issues_on_project_id_and_confidential", :algorithm=>:concurrently})
main:    -> 0.0001s
main: == 20220602111923 PrepareIndexForOpenIssuesCount: migrated (0.0177s) ==========

ci: == 20220602111923 PrepareIndexForOpenIssuesCount: migrating ===================
ci: -- index_exists?(:issues, [:project_id, :confidential], {:where=>"state_id = 1", :name=>"idx_open_issues_on_project_id_and_confidential", :algorithm=>:concurrently})
ci:    -> 0.0079s
ci: -- add_index_options(:issues, [:project_id, :confidential], {:where=>"state_id = 1", :name=>"idx_open_issues_on_project_id_and_confidential", :algorithm=>:concurrently})
ci:    -> 0.0001s
ci: == 20220602111923 PrepareIndexForOpenIssuesCount: migrated (0.0118s) ==========

main: == 20220602111923 PrepareIndexForOpenIssuesCount: reverting ===================
main: == 20220602111923 PrepareIndexForOpenIssuesCount: reverted (0.0085s) ==========

ci: == 20220602111923 PrepareIndexForOpenIssuesCount: reverting ===================
ci: == 20220602111923 PrepareIndexForOpenIssuesCount: reverted (0.0085s) ==========

Related to #363603 (closed)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Heinrich Lee Yu

Merge request reports