GlobalSearchSampler is using an unaccounted for database connection
Discovered in gitlab-com/gl-infra/scalability#409 (comment 384015458)
It looks like the Gitlab::Metrics::Samplers::GlobalSearchSampler
, that runs inside sidekiq processes is using a database connection.
I suspect it's using this connection to check if elasticsearch is available in the license and enabled. Normally, this query would only happen once, since the Gitlab::CurrentSettings
are chached in the request store. But since this call-site does not pass through the request store-middlewares for sidekiq, the value will be reloaded every time the sampler is called.
I'm not very concerned about the database call itself, though it would be nice to be able to keep it out of the samplers.
We do need to account for it in Gitlab::Runtime.max_threads
. We base the number of connections in the connection pool on this number during initialization.
Proposals:
- Increment
Gitlab::Runtime.max_threads
. - Avoid the database call all together. Since the service is tracking queue sizes, would it make sense to see them roll to 0 if elasticsearch is disabled?