Optimize issues_with_health_status usage ping counter
What does this MR do?
In #223709 (closed), we noticed that a usage ping counter failed on Gitlab.com:
"counts.issues_with_health_status": -1,
This MR optimizes the min
/max
queries used by this batch counter. The min
query previously took over 30 mins in #database-lab
and is suspected to be the query that was timing out.
Testing (in #database-lab)
Before
Min issue ID - https://explain.depesz.com/s/7Gaej
SELECT MIN("issues"."id") FROM "issues" WHERE "issues"."health_status" IS NOT NULL
Time: 32.339 min
Result (cost=3202.63..3202.64 rows=1 width=4) (actual time=1940317.324..1940317.325 rows=1 loops=1)
Buffers: shared hit=977742 read=1835178 dirtied=6967
I/O Timings: read=1912378.429
InitPlan 1 (returns $0)
-> Limit (cost=0.56..3202.63 rows=1 width=4) (actual time=1940317.262..1940317.262 rows=1 loops=1)
Buffers: shared hit=977742 read=1835178 dirtied=6967
I/O Timings: read=1912378.429
-> Index Scan using issues_pkey on public.issues (cost=0.56..9651011.95 rows=3014 width=4) (actual time=1940317.259..1940317.259 rows=1 loops=1)
Index Cond: (issues.id IS NOT NULL)
Filter: (issues.health_status IS NOT NULL)
Rows Removed by Filter: 3413749
Buffers: shared hit=977742 read=1835178 dirtied=6967
I/O Timings: read=1912378.429
Max issue ID - https://explain.depesz.com/s/ZwWI
SELECT MAX("issues"."id") FROM "issues" WHERE "issues"."health_status" IS NOT NULL
Time: 2.343 s
Result (cost=3202.63..3202.64 rows=1 width=4) (actual time=2342.772..2342.773 rows=1 loops=1)
Buffers: shared hit=250 read=1562 dirtied=1303
I/O Timings: read=2264.000
InitPlan 1 (returns $0)
-> Limit (cost=0.56..3202.63 rows=1 width=4) (actual time=2342.765..2342.765 rows=1 loops=1)
Buffers: shared hit=250 read=1562 dirtied=1303
I/O Timings: read=2264.000
-> Index Scan using issues_pkey on public.issues (cost=0.56..9651011.95 rows=3014 width=4) (actual time=2342.763..2342.763 rows=1 loops=1)
Index Cond: (issues.id IS NOT NULL)
Filter: (issues.health_status IS NOT NULL)
Rows Removed by Filter: 1297
Buffers: shared hit=250 read=1562 dirtied=1303
I/O Timings: read=2264.000
After
Min issue ID - https://explain.depesz.com/s/VTrs
SELECT MIN("issues"."id") FROM "issues"
Time: 8.779 ms
Result (cost=0.59..0.60 rows=1 width=4) (actual time=8.263..8.263 rows=1 loops=1)
Buffers: shared read=5
I/O Timings: read=8.154
InitPlan 1 (returns $0)
-> Limit (cost=0.56..0.59 rows=1 width=4) (actual time=8.255..8.256 rows=1 loops=1)
Buffers: shared read=5
I/O Timings: read=8.154
-> Index Only Scan using issues_pkey on public.issues (cost=0.56..1072878.83 rows=43071125 width=4) (actual time=8.254..8.254 rows=1 loops=1)
Index Cond: (issues.id IS NOT NULL)
Heap Fetches: 0
Buffers: shared read=5
I/O Timings: read=8.154
Max issue ID - https://explain.depesz.com/s/S99Yy
SELECT MAX("issues"."id") FROM "issues"
Time: 13.320 ms
Result (cost=0.59..0.60 rows=1 width=4) (actual time=12.885..12.886 rows=1 loops=1)
Buffers: shared hit=1 read=5 dirtied=1
I/O Timings: read=12.672
InitPlan 1 (returns $0)
-> Limit (cost=0.56..0.59 rows=1 width=4) (actual time=12.878..12.879 rows=1 loops=1)
Buffers: shared hit=1 read=5 dirtied=1
I/O Timings: read=12.672
-> Index Only Scan using issues_pkey on public.issues (cost=0.56..1072878.83 rows=43071125 width=4) (actual time=12.876..12.876 rows=1 loops=1)
Index Cond: (issues.id IS NOT NULL)
Heap Fetches: 1
Buffers: shared hit=1 read=5 dirtied=1
I/O Timings: read=12.672
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
- [-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
- [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team
Closes #223709 (closed)
Edited by Alishan Ladhani