gitlab-ci-multi-runner fails to submit builds to coordinator due to permission issues
We had a working GitLab 7.14 installation with GitLab CI enabled. Did a apt-get update && apt-get dist-upgrade
and that upgraded GitLab to 8.1.2-ce.0
. New GitLab looks real cool, thanks for the good job!
We didn't need the old build info, so seeing it gone was not a big deal for us. We installed and enabled gitlab-ci-multi-runner
version 0.6.2
according to official docs with apt-get install gitlab-ci-multi-runner
and did set it up according to instructions.
However, builds remained hanging in running status because submitting builds failed with status 500.
When running the build runner in debug mode:
gitlab-ci-multi-runner stop
gitlab-ci-multi-runner --debug run
then output was as follows:
INFO[0000] f5a7637a 16 Using Shell executor...
DEBU[0000] f5a7637a 16 Starting shell command...
DEBU[0000] f5a7637a 16 Waiting for signals...
DEBU[0003] 1 Feeding runners to channel
WARN[0003] f5a7637a 16 Submitting build to coordinator... failed 500 Internal Server Error
and the runner kept retrying endlessly.
So I dug through /var/log/gitlab/gitlab-rails/production.log
and discovered the following lines:
Started PUT "/ci/api/v1/builds/16.json" for x.x.x.x at 2015-11-02 20:59:56 +0200
Errno::EACCES (Permission denied @ dir_s_mkdir - /var/opt/gitlab/gitlab-ci/builds/2015_11/14):
/opt/gitlab/embedded/lib/ruby/2.1.0/fileutils.rb:250:in `mkdir'
and indeed, /var/opt/gitlab/gitlab-ci/builds/
was owned by gitlab-ci
, but the service seems to be running as git
user.
So I solved the problem by running
chown -R git: /var/opt/gitlab/gitlab-ci/builds/
After that everything is working well.
I suggest this chown
line to be added to package upgrade/setup scripts to avoid this trouble for others.