Skip to content

Optimize service_desk_issue counters in usage data

What does this MR do?

  1. Makes the service desk issue queries non-distinct as it's wrong to count the old way
  2. Adds an index to the issues table

Issue #208888 (closed)

Optimization

CREATE INDEX index_issues_on_author_id_and_id_and_created_at ON public.issues USING btree (author_id, id, created_at);
# The query has been executed. Duration: 11.673 min (edited) 

Query 1:

explain SELECT MIN("issues"."id") FROM "issues" WHERE "issues"."author_id" = 1257257 AND "issues"."created_at" BETWEEN '2020-02-10 09:20:08.715339' AND '2020-03-09 09:20:08.715417';

Before: Bad! https://explain.depesz.com/s/ypMy was timing out

After: Good! https://explain.depesz.com/s/38XJ is now < 100ms

Query 2

SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."author_id" = 1257257 AND "issues"."created_at" BETWEEN '2020-02-10 09:20:08.715339' AND '2020-03-09 09:20:08.715417' AND "issues"."id" BETWEEN 1300000 AND 1399999

Before: Bad! https://explain.depesz.com/s/633Z => Over 40 seconds

After: Good! https://explain.depesz.com/s/65N9 => Total count time was pessimistically 2.2 seconds: calculation (1ms per loop) * 22M / 100K

Timing

issues is one of our most challenging tables with > 500million records.

Before the index, the query times out and is uncountable.

After the index, very pessimistic batch counting takes 44 minutes (with database-lab numbers) - expected duration is 20 minutes or less

  • 5.5 million users,
  • with 1250 batch sizes
  • 5.5M/1250K = 4400 loops
  • Each loop takes: 600 ms ( cold cache )

Migration output

$ VERBOSE=true bundle exec rake db:migrate:up VERSION=20200306170211
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: migrating ======
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:issues, [:author_id, :id, :created_at], {:algorithm=>:concurrently})
   -> 0.0063s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- add_index(:issues, [:author_id, :id, :created_at], {:algorithm=>:concurrently})
   -> 0.0225s
-- execute("RESET ALL")
   -> 0.0004s
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: migrated (0.0298s) 

$ VERBOSE=true bundle exec rake db:migrate:down VERSION=20200306170211
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: reverting ======
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:issues, [:author_id, :id, :created_at], {:algorithm=>:concurrently})
   -> 0.0066s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- remove_index(:issues, {:algorithm=>:concurrently, :column=>[:author_id, :id, :created_at]})
   -> 0.0238s
-- execute("RESET ALL")
   -> 0.0014s
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: reverted (0.0325s) 


Edited by Alper Akgun

Merge request reports