Add saturation metrics for Rails connection pools
In https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9198 we noticed that the number of database connections available in multithreaded environments was not sufficient for the number of threads that were being run.
ActiveRecord's connection pool provides a #stat
method for monitoring just this kind of thing: https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html#method-i-stat
We'll record metrics corresponding to the #stat
method. These will be per:
- Node.
- Database class (we have three sources of database connection pools:
ActiveRecord::Base
,Geo::TrackingBase
, andGitlab::Database::LoadBalancing::Host
). - Database host and port. (On GitLab.com we have load balancing set up to use the same hosts on different ports, so the host isn't sufficient for a unique identification.)
We'll also set a saturation metric as described in #153 (comment 337651966):
ActiveRecord::Base.connection_pool.stat.busy + ActiveRecord::Base.connection_pool.stat.dead
/
ActiveRecord::Base.connection_pool.stat.size
So that we can monitor this going forwards.