Gitaly nodes not reloaded on pre and staging
Validating gitaly version on pre, we found that it was running a very old version 1.67.0
instead of 1.72.0-14-gba4abcb7
.
Digging into the logs we found that the graceful upgrade process timed out (1 min) in the attempt to spawn a child process.
We were able to reproduce this with gitlab-ctl hup gitaly
Restarting the process with gitlab-ctl restart gitaly
worked as expected, but now the process seems to ignore other HUP signals.
It fails with the error message Process not ready to start an upgrade
(TODO: check the right message)
We had the same exact behavior on staging.
Production had a successful graceful upgrade and is running the expected version.