gitlab-ci-multi-runner fails to submit builds to coordinator due to permission issues
We had a working GitLab 7.14 installation with GitLab CI enabled. Did a
apt-get update && apt-get dist-upgrade and that upgraded GitLab to
8.1.2-ce.0. New GitLab looks real cool, thanks for the good job!
We didn't need the old build info, so seeing it gone was not a big deal for us. We installed and enabled
0.6.2 according to official docs with
apt-get install gitlab-ci-multi-runner and did set it up according to instructions.
However, builds remained hanging in running status because submitting builds failed with status 500.
When running the build runner in debug mode:
gitlab-ci-multi-runner stop gitlab-ci-multi-runner --debug run
then output was as follows:
INFO f5a7637a 16 Using Shell executor... DEBU f5a7637a 16 Starting shell command... DEBU f5a7637a 16 Waiting for signals... DEBU 1 Feeding runners to channel WARN f5a7637a 16 Submitting build to coordinator... failed 500 Internal Server Error
and the runner kept retrying endlessly.
So I dug through
/var/log/gitlab/gitlab-rails/production.log and discovered the following lines:
Started PUT "/ci/api/v1/builds/16.json" for x.x.x.x at 2015-11-02 20:59:56 +0200 Errno::EACCES (Permission denied @ dir_s_mkdir - /var/opt/gitlab/gitlab-ci/builds/2015_11/14): /opt/gitlab/embedded/lib/ruby/2.1.0/fileutils.rb:250:in `mkdir'
/var/opt/gitlab/gitlab-ci/builds/ was owned by
gitlab-ci, but the service seems to be running as
So I solved the problem by running
chown -R git: /var/opt/gitlab/gitlab-ci/builds/
After that everything is working well.
I suggest this
chown line to be added to package upgrade/setup scripts to avoid this trouble for others.