Skip to content

Add saturation metrics for Rails connection pools

In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9198 we noticed that the number of database connections available in multithreaded environments was not sufficient for the number of threads that were being run.

ActiveRecord's connection pool provides a #stat method for monitoring just this kind of thing: https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html#method-i-stat

We'll record metrics corresponding to the #stat method. These will be per:

  1. Node.
  2. Database class (we have three sources of database connection pools: ActiveRecord::Base, Geo::TrackingBase, and Gitlab::Database::LoadBalancing::Host).
  3. Database host and port. (On GitLab.com we have load balancing set up to use the same hosts on different ports, so the host isn't sufficient for a unique identification.)

We'll also set a saturation metric as described in #153 (comment 337651966):

ActiveRecord::Base.connection_pool.stat.busy + ActiveRecord::Base.connection_pool.stat.dead
/
ActiveRecord::Base.connection_pool.stat.size 

So that we can monitor this going forwards.

Edited by Sean McGivern