Implement strategy to remove invalid connections from pool
The connection pooling mechanism implemented in internal/connection.Pool
currently doesn't cope well in case a connection is created inside it that's invalid. One reason for it being invalid could be a wrong address, a second could be that the authentication token is invalid. While we could detect the former during creation of the connection by making the Dial()
call blocking, we can only detect invalid authentication tokens at time of first use. This is due to the fact that connection establishment itself in unauthenticated, but calling RPCs is authenticated. So we cannot detect failures here previous to calling the first RPC.
Given above, we need to be able to come up with a strategy to handle such broken connections, e.g. by evicting them from the pool or finding a way to just not add them in the first place.