systemd kills leftover kill jobs upon ssh logout on GitLab server
I'm running an Ubuntu 18.04 server to host my own GitLab installation. Until the day 10/20/2019 everything worked quite smoothly. But from that day on I started getting the following kind of messages in system log:
Oct 21 11:36:02 atlas sshd[8680]: Disconnected from user gitlab 12.34.56.78 port 60334
Oct 21 11:36:02 atlas systemd[1]: user@999.service: Killing process 8716 (kill) with signal SIGKILL.
From then on, the messages have been very frequent, typically twice per hour for every host that runs gitlab-runner, which is about twenty times per hour for me.
After some digging, I've come to the conclusion that there is a regular half-hourly ssh login on the server, and upon logout, the Linux service systemd kills leftover jobs from this ssh login. While this is standard behavior for systemd, for some reason, there are always leftover kill jobs for every GitLab ssh login. These leftover kill jobs seem not to terminate quickly enough and are in turn themselves killed by systemd. While all this may not actually be a problem, it still seems like a possible reason for concern because something in the user session is not ending cleanly. It would be great if this could be investigated and/or fixed.
I have found a workaround, to allow the gitlab user to spawn jobs that survive logout. This can be achieved for example using logind:
sudo loginctl enable-linger gitlab
Afterwards, there a no longer any log messages about killed jobs, and there are also no leftover session jobs, so this seems to lead to clean termination of the session.