Skip to content

Runner 17.5 on Local RHEL 9 hosts are failing with error: "FATAL: failed to get user home dir: $HOME is not defined"

Summary

We have a few Runners on-prem that run RedHat 9. These hosts recently upgraded to GitLab-Runner version 17.5 using the yum repository. When gitlab-runner upgraded, it no longer starts, and is saying the $HOME variable is not defined.

Steps to reproduce

On RHEL 9, upgrade the gitlab-runner to version 17.5 using the official yum repository.

Actual behavior

When I run systemctl restart gitlab-runner, it fails to start.

Expected behavior

I expect to start normally.

Relevant logs and/or screenshots

The /home/gitlab-runner/ folder is empty (not even .bashrc).

In the /var/log/messages I see the following:

Oct 17 15:55:59 [masked] gitlab-runner[254691]: #033[31;1mFATAL: failed to get user home dir: $HOME is not defined#033[0;m
Oct 17 15:55:59 [masked] systemd[1]: gitlab-runner.service: Main process exited, code=exited, status=1/FAILURE
Oct 17 15:56:00 [masked] systemd[1]: gitlab-runner.service: Failed with result 'exit-code'.

Environment description

  • Our gitLab server is a single on-prem instance.
  • Our runners are also on-pem.
  • SeLinux is running on them, but I've ran setenforce 0, and it still happens.
`config.toml` contents: 
concurrent = 4
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "[masked]"
  url = "https://[masked]/"
  id = 35
  token = "glrt-[masked]"
  token_obtained_at = 2024-09-03T21:04:38Z
  token_expires_at = 0001-01-01T00:00:00Z
  tls-ca-file = "/etc/gitlab-runner/certs/[masked].crt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0
The yum repository contents: 
[runner_gitlab-runner]
name=runner_gitlab-runner
baseurl=https://packages.gitlab.com/runner/gitlab-runner/el/9/$basearch
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://packages.gitlab.com/runner/gitlab-runner/gpgkey
       https://packages.gitlab.com/runner/gitlab-runner/gpgkey/runner-gitlab-runner-4C80FB51394521E9.pub.gpg
       https://packages.gitlab.com/runner/gitlab-runner/gpgkey/runner-gitlab-runner-49F16C5CC3A0F81F.pub.gpg
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300

[runner_gitlab-runner-source]
name=runner_gitlab-runner-source
baseurl=https://packages.gitlab.com/runner/gitlab-runner/el/9/SRPMS
repo_gpgcheck=1
gpgcheck=1
enabled=1
gpgkey=https://packages.gitlab.com/runner/gitlab-runner/gpgkey
       https://packages.gitlab.com/runner/gitlab-runner/gpgkey/runner-gitlab-runner-4C80FB51394521E9.pub.gpg
       https://packages.gitlab.com/runner/gitlab-runner/gpgkey/runner-gitlab-runner-49F16C5CC3A0F81F.pub.gpg
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
metadata_expire=300
The `/etc/systemd/system/gitlab-runner.service` contents: 
[Unit]
Description=GitLab Runner
ConditionFileIsExecutable=/usr/bin/gitlab-runner

After=network.target

[Service]
StartLimitInterval=5
StartLimitBurst=10
ExecStart=/usr/bin/gitlab-runner "run" "--config" "/etc/gitlab-runner/config.toml" "--working-directory" "/home/gitlab-runner" "--service" "gitlab-runner" "--user" "gitlab-runner"

Restart=always

RestartSec=120
EnvironmentFile=-/etc/sysconfig/gitlab-runner

[Install]
WantedBy=multi-user.target

Used GitLab Runner version

Upgraded from 17.4.1 to 17.5

# gitlab-runner --version
Version:      17.5.0
Git revision: 66a723c3
Git branch:   17-5-stable
GO version:   go1.22.7
Built:        2024-10-17T15:26:50+0000
OS/Arch:      linux/amd64

Possible fixes

The Environment File for the service may be missing. In our /etc/systemd/system/gitlab-runner.service it is set to EnvironmentFile=-/etc/sysconfig/gitlab-runner. And /etc/sysconfig/gitlab-runner was missing.

Ensure the Environment file exists and has HOME defined.

In my case, this command creates the file, sets the HOME variable, restarts the runner, and shows the status for verification.

echo "HOME=/home/gitlab-runner" >> /etc/sysconfig/gitlab-runner && \
systemctl restart gitlab-runner && \
systemctl status gitlab-runner
Edited by Brian French