Skip to content

Add partial index to improve mirrors update

What does this MR do?

Improves slow (16-18s) query produced by UpdateAllMirrorsWorker.
More at #325503 (closed)

The main offenders were:

  1. Filtering on top of the index by the status in project_mirror_data table

Postgres.ai reports

After reset: Without any changes

https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/2972/commands/9651

depesz + postgres.ai doesn't work me, so here's the manual paste: https://explain.depesz.com/s/kPSG

Screenshot_2021-03-25_at_17.22.51

reset__master

After reset: With the addition the index on project_mirror_data

https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/2972/commands/9665

Screenshot_2021-03-25_at_11.41.53

After warm-up*: Without any changes

https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/2972/commands/9661

warmup__master

After warm-up*: With the addition the index on project_mirror_data

https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/2972/commands/9667

Screenshot_2021-03-25_at_11.47.40


* warm-up = reset + run the target query multiple times in a row

The output of both migrating and rolling back for all migrations

be rake db:migrate

== 20210324112439 AddIndexMirrorDataOnRetryNextExecutionWhereStatus: migrating
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:project_mirror_data, [:next_execution_timestamp, :retry_count], {:where=>"(status)::text <> ALL ('{scheduled,started}'::text[])", :name=>"index_mirror_data_non_scheduled_or_started", :algorithm=>:concurrently})
   -> 0.0032s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- add_index(:project_mirror_data, [:next_execution_timestamp, :retry_count], {:where=>"(status)::text <> ALL ('{scheduled,started}'::text[])", :name=>"index_mirror_data_non_scheduled_or_started", :algorithm=>:concurrently})
   -> 0.0107s
-- execute("RESET ALL")
   -> 0.0004s
== 20210324112439 AddIndexMirrorDataOnRetryNextExecutionWhereStatus: migrated (0.0157s)

be rake db:rollback

== 20210324112439 AddIndexMirrorDataOnRetryNextExecutionWhereStatus: reverting
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:project_mirror_data, [:next_execution_timestamp, :retry_count], {:where=>"(status)::text <> ALL ('{scheduled,started}'::text[])", :name=>"index_mirror_data_non_scheduled_or_started", :algorithm=>:concurrently})
   -> 0.0035s
-- execute("SET statement_timeout TO 0")
   -> 0.0005s
-- remove_index(:project_mirror_data, {:where=>"(status)::text <> ALL ('{scheduled,started}'::text[])", :name=>"index_mirror_data_non_scheduled_or_started", :algorithm=>:concurrently, :column=>[:next_execution_timestamp, :retry_count]})
   -> 0.0068s
-- execute("RESET ALL")
   -> 0.0005s
== 20210324112439 AddIndexMirrorDataOnRetryNextExecutionWhereStatus: reverted (0.0121s)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Run the migration locally & rollbacked.

Run the index addition on postgres.ai (database-lab channel) and verified the query plans.

Security

N/A

Related to #325503 (closed)

Edited by Aleksei Lipniagov

Merge request reports