Gitaly does not listen on new address after SIGHUP when the previous address used the same port
Occurs with:
- gitaly['listen_addr']
- praefect['listen_addr']
- gitaly['prometheus_listen_addr']
- praefect['prometheus_listen_addr']
Steps to reproduce:
- Using Omnibus, update
gitaly['prometheus_listen_addr']
to0.0.0.0:9236
- Execute
gitlab-ctl reconfigure
, which sends aSIGHUP
to Gitaly - Prometheus will continue to listen on
localhost:9236
instead of using the new address - Gitaly logs
"error":"unable to start the bootstrap: can't create new listener: listen tcp 0.0.0.0:9236: bind: address already in use"
- Using
gitlab-ctl restart
, which sendsSIGTERM
, will cause Prometheus to successfully start listening on the new address
Note that this does not occur when changing the port as well as the address, e.g. localhost:9236
-> 0.0.0.0:6329
In strace it appears that the old listener is still up on localhost:9236
when the new socket is created. The bind
call for 0.0.0.0:9236
fails with EADDRINUSE
, despite SO_REUSEADDR
being set. Per this SO answer, setting SO_REUSEPORT
would probably resolve the error.
Edited by Will Chandler