Investigate cause of WebsocketsServiceRailsRequestApdexSLOViolationRegional alerts
Summary
Apdex for websockets consistently dropping below SLO.
This corresponds with a large number of 500 errors for ActionCable::Connection with 'can't alloc thread'.
Impact
This resulted in multiple incidents
- production#15977 (closed)
- production#15975 (closed)
- production#16060 (closed)
- production#16103 (closed)
- production#16130 (closed)
- production#16128 (closed)
- production#16144 (closed)
- production#16146 (closed)
- production#16168 (closed)
Recommendation
Verification
Confirm that the 500s have stopped and that the apdex has stopped dropping below SLO.