Gracefully stop long running processes when using the shell executor - Take 2

This is a second attempt at Gracefully stop long running processes when usi... (!4601 - merged) which was reverted in Revert "Merge branch 'avonbertoldi/27443/gracef... (!4686 - merged) because it broke @tmaczukin s**t.

Compared to Gracefully stop long running processes when usi... (!4601 - merged), instead of completely changing the meaning of the --user arg to gitlab-runner install, this approach adds an --init-user arg which is mutually exclusive with --user, and will do what --user did in Gracefully stop long running processes when usi... (!4601 - merged), which is...

When --init-user is specified instead of --user in the gitlab-runner install invocation, the generated unit file will change like so:

- ExecStart=/usr/bin/gitlab-runner "run" "--config" "/etc/gitlab-runner/config.toml" "--working-directory" "/home/gitlab-runner" "--service" "gitlab-runner" "--user" "gitlab-runner"
+ ExecStart=/usr/bin/gitlab-runner "run" "--config" "/home/gitlab-runner/config.toml" "--working-directory" "/home/gitlab-runner" "--service" "gitlab-runner"
+ User=gitlab-runner

and config.toml is copied from /etc/gitlab-runner to /home/gitlab-runner if it does not exist at the destination.

Recall from Gracefully stop long running processes when usi... (!4601 - merged) that, for the shell executor only, when --user is included in the gitlab-runner run invocation, scripts are run using su to assume the specified users identity. This fact alone prevents graceful shutdown of long-running processes, and indeed stopping of such processes altogether. This is because when a job is cancelled or times out, SIGTERM is sent to the su process, which exits, leaving orphaned it's entire child process tree. --user is added to the run command when also specified in install command.

The fix is to specify the user under which to run the gitlab-runner service in the unit file itself, and to not specify --user in gitlab-runner run. This cuts su out of the picture, and allows signal forwarding and process reaping to happen normally, enabling graceful process tree shutdown when jobs time-out or are canceled.

To enable the gitlab-runner install --init-user option to be specified during package installation, I'm relying on the USE_INIT_USER environment variable.

I've tested this on debian with the following results:

No $USE_INIT_USER

Unit file

[Unit]
Description=GitLab Runner
ConditionFileIsExecutable=/usr/bin/gitlab-runner
After=network.target 
[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/bin/gitlab-runner "run" "--config" "/etc/gitlab-runner/config.toml" "--working-directory" "/home/gitlab-runner" "--service" "gitlab-runner" "--user" "gitlab-runner"
Restart=always
RestartSec=120
EnvironmentFile=-/etc/sysconfig/gitlab-runner
[Install]
WantedBy=multi-user.target

And new skeleton config.toml created at /etc/gitlab-runner/config.toml

With $USE_INIT_USER

Unit file

[Unit]
Description=GitLab Runner
ConditionFileIsExecutable=/usr/bin/gitlab-runner
After=network.target 
[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/bin/gitlab-runner "run" "--config" "/home/gitlab-runner/config.toml" "--working-directory" "/home/gitlab-runner" "--service" "gitlab-runner"
User=gitlab-runner
Restart=always
RestartSec=120
EnvironmentFile=-/etc/sysconfig/gitlab-runner
[Install]
WantedBy=multi-user.target

And new skeleton config.toml created at /etc/gitlab-runner/config.toml and copied to /home/gitlab-runner/config.toml (and permissions updated).

In both cases systemctl status gitlab-runner.service indicated the service was running correctly.

Closes #37000 (closed)