Gitaly does not listen on new address after SIGHUP when the previous address used the same port
Occurs with:
- gitaly['listen_addr']
- praefect['listen_addr']
- gitaly['prometheus_listen_addr']
- praefect['prometheus_listen_addr']
Steps to reproduce:
- Using Omnibus, update
gitaly['prometheus_listen_addr']to0.0.0.0:9236 - Execute
gitlab-ctl reconfigure, which sends aSIGHUPto Gitaly - Prometheus will continue to listen on
localhost:9236instead of using the new address - Gitaly logs
"error":"unable to start the bootstrap: can't create new listener: listen tcp 0.0.0.0:9236: bind: address already in use" - Using
gitlab-ctl restart, which sendsSIGTERM, will cause Prometheus to successfully start listening on the new address
Note that this does not occur when changing the port as well as the address, e.g. localhost:9236 -> 0.0.0.0:6329
In strace it appears that the old listener is still up on localhost:9236 when the new socket is created. The bind call for 0.0.0.0:9236 fails with EADDRINUSE, despite SO_REUSEADDR being set. Per this SO answer, setting SO_REUSEPORT would probably resolve the error.
Edited by Will Chandler (ex-GitLab)