Skip to content

Add instrumentation for Redis connection pools

Sylvester Chin requested to merge sc1-instrument-connection-pool into master

What does this MR do and why?

This MR adds instrumentation for connection pools (the main purpose is to observe Redis connection pools).

The 2 gauges gitlab_connection_pool_size and gitlab_connection_pool_available_count will help track the utilisation of connection pools. For sidekiq, the connection pool pool_name will be "unknown" for Sidekiq 6.x.

In Sidekiq 7.x, pools are named:

See related issue: gitlab-com/gl-infra/scalability#2564 (comment 1672791437). This closes gitlab-com/gl-infra/scalability#2649 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. In rails console, run a thread
Thread.new do
   Gitlab::Redis::Queues.with do |r|
     r
     sleep(1000)
   end
end
  1. Check the pool size:
require 'prometheus/client/formats/text.rb'
Prometheus::Client::Formats::Text.marshal_multiprocess.split("\n").filter{|x| x.include?("gitlab_connection_pool_available") && x.include?(Process.pid) }
"gitlab_connection_pool_available_count{pool_key=\"pool-177440\",pool_name=\"queues\",pid=\"process_45289\"} 4"]
  1. Checkout another connection and check that the available count is 3.
Gitlab::Redis::Queues.with { |c| c.ping }
Prometheus::Client::Formats::Text.marshal_multiprocess.split("\n").filter{|x| x.include?("gitlab_connection_pool_available") && x.include?(Process.pid) }

There should be a line like the following, since 2 connections are checked out. One is still held by the pool, and the gauge was set to 3 when Gitlab::Redis::Queues.with { |c| c.ping } was run.

"gitlab_connection_pool_available_count{pool_key=\"pool-177440\",pool_name=\"queues\",pid=\"process_45289\"} 3"]

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports