Skip to content

Optimize projects_service_active queries

What does this MR do?

Optimizes the projects_*_active counter queries for batch counting in usage data

Part of issue #208923 (closed)

Query

SELECT COUNT(services.id) FROM services WHERE services.type = JiraService AND services.active = true AND services.template = false AND services.id BETWEEN 0 AND 9999;

Optimization

CREATE CONCURRENT INDEX index_services_on_type_and_id_and_template ON public.services USING btree (type, id, template) WHERE (active = true);
-- The query has been executed. Duration: 10.803 s

Before: Bad! https://explain.depesz.com/s/FsVe

After: Good! https://explain.depesz.com/s/CqcL

Timing

After the index for batch counting takes 1 second pessimistic

  • ~3.6M services,
  • with 10_000 batch sizes
  • 3.6M/10_000 = 360 loops
  • Time: < 0.5 ms ( cold cache with no time constraint )

Migration output

VERBOSE=true bundle exec rake db:migrate:up VERSION=20200312125121
== 20200312125121 AddIndexOnActiveAndTemplateAndTypeAndIdToServices: migrating
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:services, [:type, :id, :template], {:where=>"active = TRUE", :algorithm=>:concurrently})
   -> 0.0033s
-- execute("SET statement_timeout TO 0")
   -> 0.0001s
-- add_index(:services, [:type, :id, :template], {:where=>"active = TRUE", :algorithm=>:concurrently})
   -> 0.0032s
-- execute("RESET ALL")
   -> 0.0001s
== 20200312125121 AddIndexOnActiveAndTemplateAndTypeAndIdToServices: migrated (0.0069s)

VERBOSE=true bundle exec rake db:migrate:down VERSION=20200312125121
== 20200312125121 AddIndexOnActiveAndTemplateAndTypeAndIdToServices: reverting
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:services, [:type, :id, :template], {:where=>"active = TRUE", :algorithm=>:concurrently})
   -> 0.0035s
-- execute("SET statement_timeout TO 0")
   -> 0.0001s
-- remove_index(:services, {:where=>"active = TRUE", :algorithm=>:concurrently, :column=>[:type, :id, :template]})
   -> 0.0046s
-- execute("RESET ALL")
   -> 0.0002s
== 20200312125121 AddIndexOnActiveAndTemplateAndTypeAndIdToServices: reverted (0.0085s)
Edited by Alex Buijs

Merge request reports