Gitlab::Memory::Watchdog is holding a database connection forever
Problem
As discovered in #423945 (comment 1544795423) when the Gitlab::Memory::Watchdog
starts up it creates a Thread that runs forever. Once it executes any database query in this thread it will checkout a connection from a connection pool. It is doing this to check Feature.enabled?.
Keeping a database connection checked out has been causing our disconnects to timeout when we do DB service discovery. Every time we change replica hosts from service discovery we need to disconnect old hosts. But the problem is we aren't able to disconnect old hosts when they have a checked out connection.
Solution
Do not query the database in background threads or at the very least you need to check the connection back in when you are done with it. Perhaps we have other cases where we do this in the application and I know that we do usually have to account for these extra background threads in our thread pool count. But something we maybe never accounted for is that replica connection pools need to be disconnected periodically and thus we need to check the connections back in.