Jetty threads leak if future that disconnects from device times out
When disconnecting from a large number of devices, the disconnect futures may time out or may not get a chance to run at all. When this happens, the underlying Jetty QueuedThreadPools do not get stopped and their threads keep running.
Details
On disconnect from device
-
SdcRemoteDevicesConnectorImpl
receivesWatchdogMessage
andonConnectionLoss()
is called -> Consumer ExecutorService is used to stopSdcRemoteDeviceImpl
-
SdcRemoteDeviceImpl.shutDown()
eventually callsEventSinkImpl.unsubscribeAll()
-
EventSinkImpl.unsubscribeAll()
uses NetworkJobThreadPool to get unsubscribe Futures withMAX_WAIT_FOR_FUTURES
timeout
Workaround
- Increase
DpwsConfig.MAX_WAIT_FOR_FUTURES
, so the disconnect futures do not time out - Increase the thread count for
Consumer Executor
inGlueModule
Edited by Tiberiu Viulet