error: could not lock config file //.gitconfig: Permission denied if containers are executed as a different user than root

Workaround

A workaround is explained in this comment 👉🏿 #37408 (comment 2220748899)

Summary

The Helper runs at the beginning of a run might fail with the error:

error: could not lock config file //.gitconfig: Permission denied

This has been reported in the past:

• #32145 (closed)
• #3511 (closed)

However, mostly fixed with deep in the ways how runners are deployen on OpenShift etc. and it can actually be also easily reproduced on regular docker runners.

Steps to reproduce

We do configure one of our docker runners (backed by rootless podman) with the following:

  userns_mode = "keep-id:uid=1000,gid=1000"

This has the intention, that we map the outside UID into uid 1000 within the container and run processes within containers with uid 1000 (if not specified differently).

As soon as we start configuring that, we do see the above mentioned error.

Actual behavior

Gilab runner helper fails to setup git config properly and thus also to clone the repository into the runner.

Expected behavior

Cloning just works also with a different user.

Relevant logs and/or screenshots

There is not more logs than

error: could not lock config file //.gitconfig: Permission denied

even if you start the job runs with debug output.

The issue comes clearly, because the uid the helper scripts are executed as within the helper container, do not have a /etc/passwd enty and thus do not have HOME set as env. Therefor one of the first calls to git config --global within the helper scripts cannot really create a proper path to the global .gitconfig and thus tries to write it to //.gitconfig (already the double slashes in the path give you the indication that setting up the path failed with an empty variable - $HOME in that case).

As mentioned in other linked issues. Setting the HOME ENV variable seem to have helped, but not in our case.

I would argue that the helper script should be more robust in setting up the env, and for example create a HOME dir if not present and then set an own ENV variable to that directory, so all the git commands can use the global config from a writable HOME directory.

Environment description

Latest gitlab-runner (installed on EL 9 through official RPMs), with the gitlab-runner-helper images running as docker runner through a rootless podman with a docker socket exposed.

Used GitLab Runner version

$ gitlab-runner --version
Version:      15.4.0
Git revision: 43b2dc3d
Git branch:   refs/pipelines/647216591
GO version:   go1.17.9
Built:        2022-09-21T22:15:43+0000
OS/Arch:      linux/amd64

Possible fixes

A workaround we found is to have our own runner helper extended with a user:

FROM registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest

RUN adduser -D -u 1000 user

Which provides enough /etc/passwd context for the scripts to work.

But I do think the helper script should not assume a correctly setup user env as mentioned above.

If you do not think this should be improved and the helper script should handle such a situation, I would say the helper script must be improved towards a state, where it gives you a proper error message, e.g like:

The HOME directory of the current user (set to '<value of $HOME>') is not writable, cannot configure git correctly and therefor abort any further execution. Please make sure your helper environment is correctly setup.