Optimize ci_builds counters for non MAU
We add an index to optimize ci_jobs with a name.
- WHERE
"ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = '?'
- Where ? are the security parameters
[:container_scanning, :dast, :dependency_scanning, :license_management, :license_scanning, :sast]
Main issue #208884 (comment 310936393)
Optimization
CREATE INDEX index_ci_builds_on_name_and_security_type_eq_ci_build ON public.ci_builds USING btree (name, id) WHERE ((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text, ('dast'::character varying)::text, ('dependency_scanning'::character varying)::text, ('license_management'::character varying)::text, ('sast'::character varying)::text, ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)
# Non-concurrent build time. Duration: 115.383 min (edited)
# Concurrent build The query has been executed. Duration: 167.055 min (edited)
Query 1
SELECT MIN("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'container_scanning'
https://explain.depesz.com/s/jESy
Before: It takes too much time in database-lab, query plan on local laptop ishttps://explain.depesz.com/s/gmyx < 10 msec
After:Query 2
SELECT MAX("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'container_scanning'
https://explain.depesz.com/s/Wt4i takes 1.2 seconds
Before:https://explain.depesz.com/s/5MPm < 10 msec
After:Query 3
SELECT COUNT("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'container_scanning' AND "ci_builds"."id" BETWEEN 446900000 AND 446999999
https://explain.depesz.com/s/Rmr ~ 4.4 seconds
Before:https://explain.depesz.com/s/27Ko < 10.msec
After:Count timing
- 5.5 million users / 1250 => 4400 loops with < 10 msecs => 44 seconds
Migration Output
$ VERBOSE=true bundle exec rake db:migrate:up VERSION=20200325160952
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: migrating ============
-- transaction_open?()
-> 0.0000s
-- index_exists?(:ci_builds, [:name, :id], {:name=>"index_ci_builds_on_name_and_security_type_eq_ci_build", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text, ('dast'::character varying)::text, ('dependency_scanning'::character varying)::text, ('license_management'::character varying)::text, ('sast'::character varying)::text, ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
-> 0.0074s
-- execute("SET statement_timeout TO 0")
-> 0.0001s
-- add_index(:ci_builds, [:name, :id], {:name=>"index_ci_builds_on_name_and_security_type_eq_ci_build", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text, ('dast'::character varying)::text, ('dependency_scanning'::character varying)::text, ('license_management'::character varying)::text, ('sast'::character varying)::text, ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
-> 0.0118s
-- execute("RESET ALL")
-> 0.0006s
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: migrated (0.0202s) ===
$ VERBOSE=true bundle exec rake db:migrate:down VERSION=20200325160952
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: reverting ============
-- transaction_open?()
-> 0.0000s
-- indexes(:ci_builds)
-> 0.0082s
-- execute("SET statement_timeout TO 0")
-> 0.0001s
-- remove_index(:ci_builds, {:algorithm=>:concurrently, :name=>"index_ci_builds_on_name_and_security_type_eq_ci_build"})
-> 0.0100s
-- execute("RESET ALL")
-> 0.0002s
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: reverted (0.0187s) ===
Edited by Alper Akgun