You need to sign in or sign up before continuing.

Runner authentication failed

Summary

Installed a GitLab Runner today on a new Debian Jessie machine, from the provided apt repository. Registering the runner works, and it seems happy (status, list and verify are OK). No pipelines ever work, though, as the runner fails to authenticate to GitLab during the attempted git clone (using simplest case of the shell executor with a trivial test script). Note that the custom GitLab instance (on another machine, using the provided GitLab CE packages, omnibus) is protected with OmniAuth Shibboleth and runs within Apache httpd using mod_passenger (to avoid HTTP proxying to Nginx), which may well account for this error and makes this hard to reproduce elsewhere, I guess.

Steps to reproduce

Set up a runner for a project, use the most trivial CI yaml ever, make a commit to the repo, see it fail. I guess one would need a similar environment, as there's nothing special about the runner, the repo, the CI or the network connection to GitLab? 😞

Actual behavior

Build fails due to failed authntication to custom GitLab instace. HTTP and HTTPS connections from the runner to the GitLab instance are fine.

Expected behavior

Git clone works and build starts.

Personally I'd much rather give runners deploy keys and have them use SSH for cloning (i.e., all repo access), as that would certainly avoid this error -- git clone via SSH works fine from the machine with the runner.

Relevant logs and/or screenshots

From the "pipeline failed" notification email (which was surprisingly detailed, so quite helpful, IMO):

Stage: test                                                                                                                                                                                                                        
Name: what_python                                                                                                                                                                                                                  
Trace: ^[[0KRunning with gitlab-ci-multi-runner 1.8.1 (a2efdd4)                                                                                                                                                                    
^[[0KUsing Shell executor...                                                                                                                                                                                                       
Running on foo-dev...                                                                                                                                                                                                            
Cloning repository...                                                                                                                                                                                                              
Cloning into '/home/gitlab-runner/builds/f1eec41c/0/somegroup/ci-test'...                                                                                                                                                           
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@git.example.org/somegroup/ci-test.git/': The requested URL returned error: 500                                                                                    
ERROR: Build failed: exit status 1

That seems to correspond to these log entries in GitLab's web server access log (httpd here):

ip.add.re.ss - - [13/Dec/2016:08:44:32 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 201 6065 "-" "gitlab-ci-multi-runner 1.8.1 (1-8-stable; go1.7.3; linux/amd64)"
ip.add.re.ss - - [13/Dec/2016:08:44:32 +0000] "GET /somegroup/ci-test.git/info/refs?service=git-upload-pack HTTP/1.1" 401 4472 "-" "git/2.1.4"
ip.add.re.ss - - [13/Dec/2016:08:44:32 +0000] "GET /somegroup/ci-test.git/info/refs?service=git-upload-pack HTTP/1.1" 500 3055 "-" "git/2.1.4"
ip.add.re.ss - - [13/Dec/2016:08:44:32 +0000] "PUT /ci/api/v1/builds/9.json HTTP/1.1" 200 4120 "-" "gitlab-ci-multi-runner 1.8.1 (1-8-stable; go1.7.3; linux/amd64)"
ip.add.re.ss - - [13/Dec/2016:08:44:35 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 201 6059 "-" "gitlab-ci-multi-runner 1.8.1 (1-8-stable; go1.7.3; linux/amd64)"
ip.add.re.ss - - [13/Dec/2016:08:44:35 +0000] "GET /somegroup/ci-test.git/info/refs?service=git-upload-pack HTTP/1.1" 401 4472 "-" "git/2.1.4"
ip.add.re.ss - - [13/Dec/2016:08:44:35 +0000] "GET /somegroup/ci-test.git/info/refs?service=git-upload-pack HTTP/1.1" 500 3055 "-" "git/2.1.4"
ip.add.re.ss - - [13/Dec/2016:08:44:35 +0000] "PUT /ci/api/v1/builds/10.json HTTP/1.1" 200 4120 "-" "gitlab-ci-multi-runner 1.8.1 (1-8-stable; go1.7.3; linux/amd64)"
ip.add.re.ss - - [13/Dec/2016:08:44:38 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 204 4000 "-" "gitlab-ci-multi-runner 1.8.1 (1-8-stable; go1.7.3; linux/amd64)"
ip.add.re.ss - - [13/Dec/2016:08:44:41 +0000] "POST /ci/api/v1/builds/register.json HTTP/1.1" 204 4000 "-" "gitlab-ci-multi-runner 1.8.1 (1-8-stable; go1.7.3; linux/amd64)"

And the last line repeats ad infinitum as long as the runner is running (and registered), even though GitLab already shows the pipeline as failed. The matching entry from GitLab's /var/log/gitlab/gitlab-rails/production.log seems to be:

Started POST "/ci/api/v1/builds/register.json" for ip.add.re.ss at 2016-12-13 08:44:32 +0000
Started GET "/somegroup/ci-test.git/info/refs?service=git-upload-pack" for ip.add.re.ss at 2016-12-13 08:44:32 +0000
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"somegroup", "project_id"=>"ci-test.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 54ms (Views: 20.0ms | ActiveRecord: 3.5ms)
Started GET "/somegroup/ci-test.git/info/refs?service=git-upload-pack" for ip.add.re.ss at 2016-12-13 08:44:32 +0000
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"somegroup", "project_id"=>"ci-test.git"}
Completed 500 Internal Server Error in 44ms (ActiveRecord: 5.4ms)

JWT::DecodeError (Nil JSON web token):
  lib/gitlab/workhorse.rb:120:in `verify_api_request!'
  app/controllers/projects/git_http_client_controller.rb:154:in `verify_workhorse_api!'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'

which is repeated twice in the log. Then the same (and only this) log line is repeated in production.log until I stop (or unregister) the runner:

Started POST "/ci/api/v1/builds/register.json" for ip.add.re.ss at 2016-12-13 08:44:38 +0000                    
Started POST "/ci/api/v1/builds/register.json" for ip.add.re.ss at 2016-12-13 08:44:41 +0000                    
Started POST "/ci/api/v1/builds/register.json" for ip.add.re.ss at 2016-12-13 08:44:44 +0000

A manual clone attempt from the machine with the runner yields these results on the command line:

git clone https://gitlab-ci-token:f....c@git.example.org/somegroup/ci-test.git/
Cloning into 'ci-test'...
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab-ci-token:f....c@git.example.org/somegroup/ci-test.git/'

which corresponds to these events in GitLab's /var/log/gitlab/gitlab-rails/production.log:

Started GET "/somegroup/ci-test.git/info/refs?service=git-upload-pack" for ip.add.re.ss at 2016-12-13 08:58:01 +0000
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"somegroup", "project_id"=>"ci-test.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 37ms (Views: 0.6ms | ActiveRecord: 4.2ms)
Started GET "/somegroup/ci-test.git/info/refs?service=git-upload-pack" for ip.add.re.ss at 2016-12-13 08:58:01 +0000
Processing by Projects::GitHttpController#info_refs as */*
  Parameters: {"service"=>"git-upload-pack", "namespace_id"=>"somegroup", "project_id"=>"ci-test.git"}
Filter chain halted as :authenticate_user rendered or redirected
Completed 401 Unauthorized in 38ms (Views: 0.4ms | ActiveRecord: 5.5ms)

Apache httpd only sees these two requests from that manual clone attempt:

ip.add.re.ss - - [13/Dec/2016:08:55:48 +0000] "GET /somegroup/ci-test.git/info/refs?service=git-upload-pack HTTP/1.1" 401 4472 "-" "git/2.1.4"
ip.add.re.ss - - [13/Dec/2016:08:55:48 +0000] "GET /somegroup/ci-test.git/info/refs?service=git-upload-pack HTTP/1.1" 401 855 "-" "git/2.1.4"

Syslog on the runner machine only has these to offer:

Dec 13 08:44:32 foo-dev gitlab-ci-multi-runner[13639]: time="2016-12-13T08:44:32Z" level=info msg="Checking for builds... received" build=9 repo_url="https://git.example.org/somegroup/ci-test.git" runner=f1eec41c
Dec 13 08:44:32 foo-dev gitlab-ci-multi-runner[13639]: time="2016-12-13T08:44:32Z" level=warning msg="Build failed: exit status 1" build=9 project=45 runner=f1eec41c
Dec 13 08:44:35 foo-dev gitlab-ci-multi-runner[13639]: time="2016-12-13T08:44:35Z" level=info msg="Checking for builds... received" build=10 repo_url="https://git.example.org/somegroup/ci-test.git" runner=f1eec41c
Dec 13 08:44:35 foo-dev gitlab-ci-multi-runner[13639]: time="2016-12-13T08:44:35Z" level=warning msg="Build failed: exit status 1" build=10 project=45 runner=f1eec41c

I've also attached a copy of the output from gitlab-ci-multi-runner --debug run (the command I found in the FAQ): runner-debug.log

N.B. I've nuked (unregistered and registered a new one) the runner once during this testing, so the runner identifiers are not consistent everywhere throughout the above. Don't let that lead you down the wrong path!

Environment description

GitLab: This is a local and up-to-date (8.14.4-ce.0) GitLab CE Omnibus install from the provided Debian packages using (as indicated above) OmniAuth Shibboleth and running within Apache httpd using mod_passenger (to avoid HTTP proxying to Nginx). I have not noted any other issues with that setup, i.e., everything else works fine AFAICT.

The executor is shell with a trivial scriptthat merely echoes the environment (or runs python --version) as a first test.

Used GitLab Runner version

# gitlab-ci-multi-runner --version
Version:      1.8.1
Git revision: a2efdd4
Git branch:   1-8-stable
GO version:   go1.7.3
Built:        Tue, 29 Nov 2016 13:53:55 +0000
OS/Arch:      linux/amd64