Changing unicorn['listen'] takes an extended period to take affect

After changing unicorn['listen'] in gitlab.rb, and running gitlab-ctl reconfigure, it can take an extended period of time for unicorn to complete reloading after receiving a HUP.

Due to the previous unicorn master still being bound, the log will show entries like:

/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/socket_helper.rb:164:in `bind': Address already in use - bind(2) for 0.0.0.0:8080 (Errno::EADDRINUSE)
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/socket_helper.rb:164:in `new_tcp_server'
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/socket_helper.rb:144:in `bind_listen'
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:241:in `listen'
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:851:in `block in bind_new_listeners!'
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:851:in `each'
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:851:in `bind_new_listeners!'
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:140:in `start'
        from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/unicorn-5.4.1/bin/unicorn:126:in `<top (required)>'
        from /opt/gitlab/embedded/bin/unicorn:23:in `load'
        from /opt/gitlab/embedded/bin/unicorn:23:in `<main>'

In testing, restarting instead of reloading has been quicker.