Skip to content

Optimize ci_builds counters for non MAU

We add an index to optimize ci_jobs with a name.

  • WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = '?'
  • Where ? are the security parameters [:container_scanning, :dast, :dependency_scanning, :license_management, :license_scanning, :sast]

Main issue #208884 (comment 310936393)

Optimization

CREATE INDEX index_ci_builds_on_name_and_security_type_eq_ci_build ON public.ci_builds USING btree (name, id) WHERE ((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text, ('dast'::character varying)::text, ('dependency_scanning'::character varying)::text, ('license_management'::character varying)::text, ('sast'::character varying)::text, ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)

# Non-concurrent build time. Duration: 115.383 min (edited) 
# Concurrent build The query has been executed. Duration: 167.055 min (edited) 

Query 1

SELECT MIN("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'container_scanning'

Before: It takes too much time in database-lab, query plan on local laptop is https://explain.depesz.com/s/jESy

After: https://explain.depesz.com/s/gmyx < 10 msec

Query 2

SELECT MAX("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'container_scanning'

Before: https://explain.depesz.com/s/Wt4i takes 1.2 seconds

After: https://explain.depesz.com/s/5MPm < 10 msec

Query 3

SELECT COUNT("ci_builds"."id") FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."name" = 'container_scanning' AND "ci_builds"."id" BETWEEN 446900000 AND 446999999

Before: https://explain.depesz.com/s/Rmr ~ 4.4 seconds

After: https://explain.depesz.com/s/27Ko < 10.msec

Count timing

  • 5.5 million users / 1250 => 4400 loops with < 10 msecs => 44 seconds

Migration Output

$ VERBOSE=true bundle exec rake db:migrate:up VERSION=20200325160952
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: migrating ============
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:ci_builds, [:name, :id], {:name=>"index_ci_builds_on_name_and_security_type_eq_ci_build", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text, ('dast'::character varying)::text, ('dependency_scanning'::character varying)::text, ('license_management'::character varying)::text, ('sast'::character varying)::text, ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
   -> 0.0074s
-- execute("SET statement_timeout TO 0")
   -> 0.0001s
-- add_index(:ci_builds, [:name, :id], {:name=>"index_ci_builds_on_name_and_security_type_eq_ci_build", :where=>"((name)::text = ANY (ARRAY[('container_scanning'::character varying)::text, ('dast'::character varying)::text, ('dependency_scanning'::character varying)::text, ('license_management'::character varying)::text, ('sast'::character varying)::text, ('license_scanning'::character varying)::text])) AND ((type)::text = 'Ci::Build'::text)", :algorithm=>:concurrently})
   -> 0.0118s
-- execute("RESET ALL")
   -> 0.0006s
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: migrated (0.0202s) ===

$ VERBOSE=true bundle exec rake db:migrate:down VERSION=20200325160952
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: reverting ============
-- transaction_open?()
   -> 0.0000s
-- indexes(:ci_builds)
   -> 0.0082s
-- execute("SET statement_timeout TO 0")
   -> 0.0001s
-- remove_index(:ci_builds, {:algorithm=>:concurrently, :name=>"index_ci_builds_on_name_and_security_type_eq_ci_build"})
   -> 0.0100s
-- execute("RESET ALL")
   -> 0.0002s
== 20200325160952 AddIndexOnNameTypeEqCiBuildToCiBuilds: reverted (0.0187s) ===
Edited by Alper Akgun

Merge request reports

Loading