Skip to content

GitLab Next

Closed
Created by Will Chandler@wchandlerDeveloper

Gitaly does not listen on new address after SIGHUP when the previous address used the same port

Occurs with:

  • gitaly['listen_addr']
  • praefect['listen_addr']
  • gitaly['prometheus_listen_addr']
  • praefect['prometheus_listen_addr']

Steps to reproduce:

  1. Using Omnibus, update gitaly['prometheus_listen_addr'] to 0.0.0.0:9236
  2. Execute gitlab-ctl reconfigure, which sends a SIGHUP to Gitaly
  3. Prometheus will continue to listen on localhost:9236 instead of using the new address
  4. Gitaly logs "error":"unable to start the bootstrap: can't create new listener: listen tcp 0.0.0.0:9236: bind: address already in use"
  5. Using gitlab-ctl restart, which sends SIGTERM, will cause Prometheus to successfully start listening on the new address

Note that this does not occur when changing the port as well as the address, e.g. localhost:9236 -> 0.0.0.0:6329

In strace it appears that the old listener is still up on localhost:9236 when the new socket is created. The bind call for 0.0.0.0:9236 fails with EADDRINUSE, despite SO_REUSEADDR being set. Per this SO answer, setting SO_REUSEPORT would probably resolve the error.

/cc @jacobvosmaer-gitlab

Edited by Will Chandler
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information