Resolve "reconfigure should restart gitlab-runsvdir when log file location(s) changed"

Closes #3190 (closed)

---

#3190 (comment 68536998)

Right now, how our code works is that, if config file inside log directory is changed, runsv is reloaded. And this works fine (try changing log prefix and a reconfigure. You can see new logs picking new prefix without any manual action). In normal cases, when log directory itself is changed, it usually means a new config file is created. So, the existing reload code should be triggered by default when changing log directory.

However, there is a possible exception case - when switching back to an existing log directory. In that case, no new config file will be created and hence reload won't be triggered. To handle that, @twk3's suggestion should be enough - reload runsv if log/run file is changed.

I gave that a try, but without luck. The reload block ran, but log directory didn't change. From the reconfigure logs, I am seeing ruby_block "reload #{params[:name]} svlogd configuration" being executed, which means an h is being written to /opt/gitlab/sv/<service name>/log/supervise/control, which is a named pipe (this should theoretically cause runit to reload). I verified this by running cat < on the pipe and I could see h was being written. Still, logs were being written to old location and no current or lock files were created in the new location.

However, manually initiating a kill -HUP to that runsv process did indeed make the desired effect.

I am not sure why writing h to supervise file works as intended with config file and not with log/run file.

---

#3190 (comment 68642905)

It looks like it does deliver a signal:

# lsof /opt/gitlab/sv/unicorn/log/supervise/control/supervise/control
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
runsv   4194 root   11r  FIFO  253,0      0t0 101319811 /opt/gitlab/sv/unicorn/log/supervise/control
runsv   4194 root   12w  FIFO  253,0      0t0 101319811 /opt/gitlab/sv/unicorn/log/supervise/control

# strace -p 4194 &
# echo h >> /opt/gitlab/sv/unicorn/log/supervise/control
read(11, "h", 1)                        = 1
kill(4195, SIGHUP)                      = 0

# ps -ef | grep 4195
root      4195  4194  0 10:44 ?        00:00:00 svlogd -tt /var/log/gitlab/unicorn

So a HUP is being delivered to svlogd.

http://smarden.org/runit/svlogd.8.html says

If svlogd is sent a HUP signal, it closes and reopens all logs, and updates their configuration according to log/config. If svlogd has trouble opening a log directory, it prints a warning, and discards this log directory. If svlogd is unable to open all log directories given at the command line, it exits with an error.

Which is different to how runsvdir handles HUP http://smarden.org/runit/runsvdir.8.html

If runsvdir receives a HUP signal, it sends a TERM signal to each runsv(8) process it is monitoring and then exits with 111.

Does this account for the difference in behaviour you're seeing?