Skip to content

Optimise `DB probe`

Kamil Trzciński requested to merge fix-db-probe into master

Fixes

This fixes two memory/CPU aspects of DB probe:

  • connection re-use across different probes where the same connection string is used
  • the type mapping is done only once
  • the TIMEOUT is set to higher value than usually used scrapping interval of (15s or 60s)

Impact

This improves execution and memory usage by:

  • about 4M-6M less of memory allocation due to type-mapping re-use on a request, this is only done once
  • ONLY performing 1.15% of slot allocations during /metrics call compared to previously (this assumes that Sidekiq probe does not have a lot of data to be scraped)
  • ONLY performing 2.25% of malloc allocations (as above)
  • about 30% execution time due to more efficient connection re-use

Memory benchmark

The memory benchmark result for the execution of #probe_all:

# after
name=prober_all             	total_allocated_objects=2737      	total_malloc_bytes=178288        	total_mallocs=803

# before
name=prober_all                         	total_allocated_objects=178631          	total_malloc_bytes=6862088              	total_mallocs=22486    

Note: Measurements are taken with https://github.com/ruby/ruby/pull/3978.

Execution benchmark

The ab (execution time):

# after
Requests per second:    8.53 [#/sec] (mean)
Time per request:       117.235 [ms] (mean)
Time per request:       117.235 [ms] (mean, across all concurrent requests)

# before
Requests per second:    6.85 [#/sec] (mean)
Time per request:       146.084 [ms] (mean)
Time per request:       146.084 [ms] (mean, across all concurrent requests)

Outcome

  • This allows us to make our GC settings significantly more aggressive without a degradation in performance.
Edited by Kamil Trzciński

Merge request reports