Skip to content
GitLab
Next
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • gitaly gitaly
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 561
    • Issues 561
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 56
    • Merge requests 56
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • gitalygitaly
  • Issues
  • #2521
Closed
Open
Issue created Mar 04, 2020 by Will Chandler@wchandlerMaintainer

Gitaly does not listen on new address after SIGHUP when the previous address used the same port

Occurs with:

  • gitaly['listen_addr']
  • praefect['listen_addr']
  • gitaly['prometheus_listen_addr']
  • praefect['prometheus_listen_addr']

Steps to reproduce:

  1. Using Omnibus, update gitaly['prometheus_listen_addr'] to 0.0.0.0:9236
  2. Execute gitlab-ctl reconfigure, which sends a SIGHUP to Gitaly
  3. Prometheus will continue to listen on localhost:9236 instead of using the new address
  4. Gitaly logs "error":"unable to start the bootstrap: can't create new listener: listen tcp 0.0.0.0:9236: bind: address already in use"
  5. Using gitlab-ctl restart, which sends SIGTERM, will cause Prometheus to successfully start listening on the new address

Note that this does not occur when changing the port as well as the address, e.g. localhost:9236 -> 0.0.0.0:6329

In strace it appears that the old listener is still up on localhost:9236 when the new socket is created. The bind call for 0.0.0.0:9236 fails with EADDRINUSE, despite SO_REUSEADDR being set. Per this SO answer, setting SO_REUSEPORT would probably resolve the error.

/cc @jacobvosmaer-gitlab

Edited Mar 05, 2020 by Will Chandler
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking