Fix shutdown errors on program close

Intro

Eventmachine in concurrent_io is preferable for its speed, but its stability has never been asserted. Some of our previous concurrent_io fixes using event machine improved the stability of concurrent_io on start-up.

We have now encountered issues when concurrent_io throws an absolute error-storm as it gets closed down. The full error paste can be found in a comment below. I haven't yet seen these issues when a driver or a socket errors/closes; they only occur on interrupt/shutdown. if someone does see them elsewhere, please comment.

Possible Improvements

• I've added Websocket/Driver behaviour which send :close to the controller before they terminate, since terminate! previously did not close the IO object.
  • However, :close also forces the controller to call terminate! on itself. Drivers and the Websocket send a :close as part of our behaviour definition and then :terminate as part of the standard behaviour to their children. This might explain dead actors throwing errors when they receive a message, but I'm doubtful it's the full story.
• Add a unique behaviour definition to the controller class which helps manage termination and restarting
• Confirm the interaction with EventMachine upon shutdown of the controller vs on shutdown of ruby process
• Confirm the order of operations on interrupt (ie. is the main selector thread shut-down before the actors?)

Errors

Here it appears that concurrent_io is trying to reset the controller and add the IO object, even as it's being terminated, which causes an error:

ERROR -- /websocket_socket: Concurrent::RejectedExecutionError (Concurrent::RejectedExecutionError)
...
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/selector/eventmachine.rb:111:in `add'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/selector/basic.rb:96:in `add!'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/controller.rb:14:in `initialize'
...
/Users/cowan/controlenvy/configs/electronic-home-showroom/drivers/controlenvy_runtime/lib/controlenvy_runtime/websocket.rb:60:in `open_connection'
/Users/cowan/controlenvy/configs/electronic-home-showroom/drivers/controlenvy_runtime/lib/controlenvy_runtime/websocket.rb:51:in `initialize'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/core.rb:157:in `build_context'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/pausing.rb:107:in `rebuild_context'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/pausing.rb:92:in `do_reset'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent-ruby-edge-0.4.1/lib-edge/concurrent/actor/behaviour/pausing.rb:59:in `reset!'
/Users/cowan/controlenvy/configs/electronic-home-showroom/drivers/controlenvy_library/lib/controlenvy_library/behaviour.rb:58:in `do_nice_reset!'
/Users/cowan/controlenvy/configs/electronic-home-showroom/drivers/controlenvy_library/lib/controlenvy_library/behaviour.rb:41:in `on_envelope'
...
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/controller.rb:87:in `block in initialize'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/listener.rb:46:in `trigger_error'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/selector/eventmachine.rb:176:in `unbind'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:1484:in `event_callback'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:203:in `release_machine'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:203:in `ensure in run'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:206:in `run'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/selector

followed by this error, in which it appears that the trigger_error method called by event_machine selector when the IO object is closed. it appears that the parent to the controller is closed before this message can be sent, throwing an error.

/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/controller.rb:87:in `block in initialize'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/listener.rb:46:in `trigger_error'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/concurrent_io-0.2.1.with.udp/lib/concurrent_io/selector/eventmachine.rb:176:in `unbind'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:1484:in `event_callback'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:203:in `release_machine'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:203:in `ensure in run'
/Users/cowan/.rvm/gems/ruby-2.5.1/gems/eventmachine-1.2.7/lib/eventmachine.rb:206:in `run'