Optimise `Sidekiq probe`

Fixes

This makes:

• To re-use Redis connection instead of opening it each time a probe is requested and leaking the connection pool

Impact

This improves execution and memory usage by:

• we are reducing slot allocations to 50% during request for prober_all (considering that DB is optimised) and Sidekiq is lightly loaded with queues
• we are reducing mallocs to 10% as above
• we are reducing sidekiq data request due to not having to open new connections
• we tune TIMEOUT to align it with a expected scraping interval to re-use connections

Memory benchmark

The memory benchmark result for the execution of #probe_all:

# after
name=prober_new                         	total_allocated_objects=32              	total_malloc_bytes=400                  	total_mallocs=8                         
name=prober_all_sidekiq_new             	total_allocated_objects=3               	total_malloc_bytes=0                    	total_mallocs=0                         
name=prober_all_sidekiq_all_probe_stats 	total_allocated_objects=473             	total_malloc_bytes=31736                	total_mallocs=115                       
name=prober_all_sidekiq_all_probe_queues	total_allocated_objects=69              	total_malloc_bytes=5544                 	total_mallocs=15                        
name=prober_all_sidekiq_all_probe_workers	total_allocated_objects=61              	total_malloc_bytes=1408                 	total_mallocs=6                         
name=prober_all_sidekiq_all_probe_retries	total_allocated_objects=94              	total_malloc_bytes=2952                 	total_mallocs=13                        
name=prober_all_sidekiq_all             	total_allocated_objects=829             	total_malloc_bytes=44416                	total_mallocs=186                       
name=prober_all                         	total_allocated_objects=898             	total_malloc_bytes=45776                	total_mallocs=204                       

# before
name=prober_new                         	total_allocated_objects=32              	total_malloc_bytes=400                  	total_mallocs=8                         
name=prober_all_sidekiq_new             	total_allocated_objects=951             	total_malloc_bytes=428072               	total_mallocs=287                       
name=prober_all_sidekiq_all_probe_stats 	total_allocated_objects=405             	total_malloc_bytes=19632                	total_mallocs=76                        
name=prober_all_sidekiq_all_probe_queues	total_allocated_objects=67              	total_malloc_bytes=5496                 	total_mallocs=15                        
name=prober_all_sidekiq_all_probe_workers	total_allocated_objects=70              	total_malloc_bytes=5512                 	total_mallocs=15                        
name=prober_all_sidekiq_all_probe_retries	total_allocated_objects=103             	total_malloc_bytes=7008                 	total_mallocs=22                        
name=prober_all_sidekiq_all             	total_allocated_objects=777             	total_malloc_bytes=40424                	total_mallocs=165                       
name=prober_all                         	total_allocated_objects=1794            	total_malloc_bytes=469856               	total_mallocs=470                       

Execution benchmark

The ab (execution time):

# after
Requests per second:    356.21 [#/sec] (mean)
Time per request:       2.807 [ms] (mean)
Time per request:       2.807 [ms] (mean, across all concurrent requests)

# before
Requests per second:    224.54 [#/sec] (mean)
Time per request:       4.453 [ms] (mean)
Time per request:       4.453 [ms] (mean, across all concurrent requests)

Outcome

• This allows us to make our GC settings significantly more aggressive without a degradation in performance.