Skip to content

Optimize issues_with_health_status usage ping counter

What does this MR do?

In #223709 (closed), we noticed that a usage ping counter failed on Gitlab.com:

"counts.issues_with_health_status": -1,

This MR optimizes the min/max queries used by this batch counter. The min query previously took over 30 mins in #database-lab and is suspected to be the query that was timing out.

Testing (in #database-lab)

Before

Min issue ID - https://explain.depesz.com/s/7Gaej
SELECT MIN("issues"."id") FROM "issues" WHERE "issues"."health_status" IS NOT NULL

Time: 32.339 min

 Result  (cost=3202.63..3202.64 rows=1 width=4) (actual time=1940317.324..1940317.325 rows=1 loops=1)
   Buffers: shared hit=977742 read=1835178 dirtied=6967
   I/O Timings: read=1912378.429
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.56..3202.63 rows=1 width=4) (actual time=1940317.262..1940317.262 rows=1 loops=1)
           Buffers: shared hit=977742 read=1835178 dirtied=6967
           I/O Timings: read=1912378.429
           ->  Index Scan using issues_pkey on public.issues  (cost=0.56..9651011.95 rows=3014 width=4) (actual time=1940317.259..1940317.259 rows=1 loops=1)
                 Index Cond: (issues.id IS NOT NULL)
                 Filter: (issues.health_status IS NOT NULL)
                 Rows Removed by Filter: 3413749
                 Buffers: shared hit=977742 read=1835178 dirtied=6967
                 I/O Timings: read=1912378.429
Max issue ID - https://explain.depesz.com/s/ZwWI
SELECT MAX("issues"."id") FROM "issues" WHERE "issues"."health_status" IS NOT NULL

Time: 2.343 s

 Result  (cost=3202.63..3202.64 rows=1 width=4) (actual time=2342.772..2342.773 rows=1 loops=1)
   Buffers: shared hit=250 read=1562 dirtied=1303
   I/O Timings: read=2264.000
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.56..3202.63 rows=1 width=4) (actual time=2342.765..2342.765 rows=1 loops=1)
           Buffers: shared hit=250 read=1562 dirtied=1303
           I/O Timings: read=2264.000
           ->  Index Scan using issues_pkey on public.issues  (cost=0.56..9651011.95 rows=3014 width=4) (actual time=2342.763..2342.763 rows=1 loops=1)
                 Index Cond: (issues.id IS NOT NULL)
                 Filter: (issues.health_status IS NOT NULL)
                 Rows Removed by Filter: 1297
                 Buffers: shared hit=250 read=1562 dirtied=1303
                 I/O Timings: read=2264.000

After

Min issue ID - https://explain.depesz.com/s/VTrs
SELECT MIN("issues"."id") FROM "issues"

Time: 8.779 ms

 Result  (cost=0.59..0.60 rows=1 width=4) (actual time=8.263..8.263 rows=1 loops=1)
   Buffers: shared read=5
   I/O Timings: read=8.154
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.56..0.59 rows=1 width=4) (actual time=8.255..8.256 rows=1 loops=1)
           Buffers: shared read=5
           I/O Timings: read=8.154
           ->  Index Only Scan using issues_pkey on public.issues  (cost=0.56..1072878.83 rows=43071125 width=4) (actual time=8.254..8.254 rows=1 loops=1)
                 Index Cond: (issues.id IS NOT NULL)
                 Heap Fetches: 0
                 Buffers: shared read=5
                 I/O Timings: read=8.154
Max issue ID - https://explain.depesz.com/s/S99Yy
SELECT MAX("issues"."id") FROM "issues"

Time: 13.320 ms

 Result  (cost=0.59..0.60 rows=1 width=4) (actual time=12.885..12.886 rows=1 loops=1)
   Buffers: shared hit=1 read=5 dirtied=1
   I/O Timings: read=12.672
   InitPlan 1 (returns $0)
     ->  Limit  (cost=0.56..0.59 rows=1 width=4) (actual time=12.878..12.879 rows=1 loops=1)
           Buffers: shared hit=1 read=5 dirtied=1
           I/O Timings: read=12.672
           ->  Index Only Scan using issues_pkey on public.issues  (cost=0.56..1072878.83 rows=43071125 width=4) (actual time=12.876..12.876 rows=1 loops=1)
                 Index Cond: (issues.id IS NOT NULL)
                 Heap Fetches: 1
                 Buffers: shared hit=1 read=5 dirtied=1
                 I/O Timings: read=12.672

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

Closes #223709 (closed)

Edited by Alishan Ladhani

Merge request reports

Loading