## Problem As discovered in https://gitlab.com/gitlab-org/gitlab/-/issues/423945#note_1544795423 when the `Gitlab::Memory::Watchdog` starts up it creates a Thread that runs forever. Once it executes any database query in this thread it will checkout a connection from a connection pool. It is doing this to [check Feature.enabled?](https://gitlab.com/gitlab-org/gitlab/-/blob/9ff760556a9f0cc23692bf462fe5d1fac06bd471/lib/gitlab/memory/watchdog.rb#L74). Keeping a database connection checked out has been causing our disconnects to timeout when we do DB service discovery. Every time we change replica hosts from service discovery we need to disconnect old hosts. But the problem is we aren't able to disconnect old hosts when they have a checked out connection. ## Solution Do not query the database in background threads or at the very least you need to check the connection back in when you are done with it. Perhaps we have other cases where we do this in the application and I know that we do usually have to account for these extra background threads in our thread pool count. But something we maybe never accounted for is that replica connection pools need to be disconnected periodically and thus we need to check the connections back in.