Periodically re-establish database connections from app tier
What does this MR do?
Related issue: #37055 (closed)
The linked issue describes scenarios where a restart of PgBouncer hosts can cause an imbalance in connections from the app servers, since the rails app uses long-lived database connections from a connection pool. In order to prevent this from becoming an issue, the app should periodically force a reconnect of its database connections, so they can be redistributed among the available PgBouncer servers.
Unfortunately, ActiveRecord doesn't provide functionality for this out of the box. This MR patches in the functionality by checking a timer value in a callback that gets called whenever a connection is checked out of the connection pool. Since ActiveRecord caches the connection per request, by using the checkout callback, the logic only runs on the initial checkout from the pool. If the logic was added higher in the stack (perhaps by overriding .connection methods), then reconnect logic would run on subsequent usages in that request, which could trigger a reconnect mid-request.
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team