Optimize service_desk_issue counters in usage data
What does this MR do?
- Makes the service desk issue queries non-distinct as it's wrong to count the old way
- Adds an index to the issues table
Issue #208888 (closed)
Optimization
CREATE INDEX index_issues_on_author_id_and_id_and_created_at ON public.issues USING btree (author_id, id, created_at);
# The query has been executed. Duration: 11.673 min (edited)
Query 1:
explain SELECT MIN("issues"."id") FROM "issues" WHERE "issues"."author_id" = 1257257 AND "issues"."created_at" BETWEEN '2020-02-10 09:20:08.715339' AND '2020-03-09 09:20:08.715417';
https://explain.depesz.com/s/ypMy was timing out
Before: Bad!https://explain.depesz.com/s/38XJ is now < 100ms
After: Good!Query 2
SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."author_id" = 1257257 AND "issues"."created_at" BETWEEN '2020-02-10 09:20:08.715339' AND '2020-03-09 09:20:08.715417' AND "issues"."id" BETWEEN 1300000 AND 1399999
https://explain.depesz.com/s/633Z => Over 40 seconds
Before: Bad!https://explain.depesz.com/s/65N9 => Total count time was pessimistically 2.2 seconds: calculation (1ms per loop) * 22M / 100K
After: Good!Timing
issues
is one of our most challenging tables with > 500million records.
Before the index, the query times out and is uncountable.
After the index, very pessimistic batch counting takes 44 minutes (with database-lab numbers) - expected duration is 20 minutes or less
- 5.5 million users,
- with 1250 batch sizes
- 5.5M/1250K = 4400 loops
- Each loop takes: 600 ms ( cold cache )
Migration output
$ VERBOSE=true bundle exec rake db:migrate:up VERSION=20200306170211
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: migrating ======
-- transaction_open?()
-> 0.0000s
-- index_exists?(:issues, [:author_id, :id, :created_at], {:algorithm=>:concurrently})
-> 0.0063s
-- execute("SET statement_timeout TO 0")
-> 0.0003s
-- add_index(:issues, [:author_id, :id, :created_at], {:algorithm=>:concurrently})
-> 0.0225s
-- execute("RESET ALL")
-> 0.0004s
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: migrated (0.0298s)
$ VERBOSE=true bundle exec rake db:migrate:down VERSION=20200306170211
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: reverting ======
-- transaction_open?()
-> 0.0000s
-- index_exists?(:issues, [:author_id, :id, :created_at], {:algorithm=>:concurrently})
-> 0.0066s
-- execute("SET statement_timeout TO 0")
-> 0.0003s
-- remove_index(:issues, {:algorithm=>:concurrently, :column=>[:author_id, :id, :created_at]})
-> 0.0238s
-- execute("RESET ALL")
-> 0.0014s
== 20200306170211 AddIndexOnAuthorIdAndIdAndCreatedAtToIssues: reverted (0.0325s)
Edited by Alper Akgun