Gitlab CI Runner is unable to clone a repository with authorized credentials.

Summary

Gitlab CI Runner on OSX is unable to clone a repository. See the following Log output from the Build Runner:

[10:58:34]: Cloning remote git repo...
Cloning into '/var/folders/sj/9hzrpc4j3pxb04q7m7r2_4x00000gn/T/d20161224-2928-1jzf6vd'...
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://git.domain.ch/namespace/certificates.git/'
[10:58:34]: Exit status: 128
[10:58:34]: Error cloning certificates repo, please make sure you have read access to the repository you want to use
[10:58:34]: Run the following command manually to make sure you're properly authenticated:
[10:58:34]: $ git clone 'https://git.domain.ch/namespace/certificates.git' '/var/folders/sj/9hzrpc4j3pxb04q7m7r2_4x00000gn/T/d20161224-2928-1jzf6vd'
[10:58:34]: 🔒  Successfully encrypted certificates repo

[!] Error cloning certificates git repo, please make sure you have access to the repository - see instructions above
An error occured when executing task 'iOS-Certificates'.
Error: Process returned non zero exit code

ERROR: Build failed: exit status 1

The problem appeared after upgrading to 8.15 and is also present in 8.15.1. I have been able to workaround this issue for now.

Steps to reproduce

Build an Xamarin.Forms project on OSX through Gitlab CI.

I also tried to manually checkout, once the problem appears, but was also unable to clone from the command line. Only after resetting permissions of the User

Sorry I don't have an idea how one would reproduce this exact behavior.

Expected behavior

The gitlab-ci-multirunner on osx should be able to clone repository using https and an authorized user.

Actual behavior

Gitlab-ci-multirunner is sometimes not able to clone a repository, see possible fixes.

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
4/2 ... ok
6/3 ... ok
6/4 ... ok
6/5 ... ok
4/6 ... ok
9/7 ... ok
9/8 ... ok
9/9 ... ok
9/10 ... ok
9/11 ... ok
7/12 ... ok
9/14 ... ok
12/15 ... ok
9/16 ... ok
9/17 ... ok
9/18 ... ok
9/20 ... repository is empty
12/21 ... ok
13/23 ... ok
9/24 ... ok
9/25 ... ok
9/26 ... ok
13/27 ... ok
13/28 ... ok
7/29 ... ok
7/31 ... ok
2/33 ... ok
15/34 ... ok
14/36 ... ok
12/37 ... ok
2/38 ... ok
16/39 ... ok
12/40 ... ok
21/43 ... ok
4/44 ... ok
22/45 ... ok
12/46 ... ok
4/48 ... ok
23/49 ... ok
7/50 ... ok
24/52 ... ok
24/53 ... ok
24/54 ... ok
7/55 ... ok
13/56 ... ok
23/59 ... ok
25/62 ... ok
25/63 ... ok
12/65 ... ok
13/66 ... ok
13/67 ... ok
26/68 ... ok
13/69 ... ok
42/70 ... ok
13/71 ... ok
13/72 ... ok
36/73 ... ok
36/74 ... ok
41/75 ... ok
42/76 ... ok
14/77 ... ok
24/79 ... ok
45/80 ... ok
42/82 ... ok
12/83 ... ok
47/84 ... ok
42/85 ... ok
13/86 ... ok
13/87 ... ok
50/89 ... ok
14/91 ... ok
50/93 ... ok
65/94 ... ok
7/95 ... ok
14/96 ... ok
47/97 ... ok
12/98 ... ok
14/99 ... ok
13/100 ... ok
69/101 ... ok
14/102 ... repository is empty
14/103 ... ok
65/104 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
        DN: CN=Jan Humbel,CN=Users,DC=ds,DC=fiveninedigital,DC=ch        sAMAccountName: jah
        DN: CN=Tobias Herrmann,CN=Users,DC=ds,DC=fiveninedigital,DC=ch   sAMAccountName: toh
        DN: CN=Michael Höhn,CN=Users,DC=ds,DC=fiveninedigital,DC=ch      sAMAccountName: mih
        DN: CN=Wall E,CN=Users,DC=ds,DC=fiveninedigital,DC=ch    sAMAccountName: wall-e
        DN: CN=Salt Admin,CN=Users,DC=ds,DC=fiveninedigital,DC=ch        sAMAccountName: salt
        DN: CN=gaucho,CN=Users,DC=ds,DC=fiveninedigital,DC=ch    sAMAccountName: gaucho

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ... 
4/2 ... yes
6/3 ... yes
6/4 ... yes
6/5 ... yes
4/6 ... yes
9/7 ... yes
9/8 ... yes
9/9 ... yes
9/10 ... yes
9/11 ... yes
7/12 ... yes
9/14 ... yes
12/15 ... yes
9/16 ... yes
9/17 ... yes
9/18 ... yes
9/20 ... yes
12/21 ... yes
13/23 ... yes
9/24 ... yes
9/25 ... yes
9/26 ... yes
13/27 ... yes
13/28 ... yes
7/29 ... yes
7/31 ... yes
2/33 ... yes
15/34 ... yes
14/36 ... yes
12/37 ... yes
2/38 ... yes
16/39 ... yes
12/40 ... yes
21/43 ... yes
4/44 ... yes
22/45 ... yes
12/46 ... yes
4/48 ... yes
23/49 ... yes
7/50 ... yes
24/52 ... yes
24/53 ... yes
24/54 ... yes
7/55 ... yes
13/56 ... yes
23/59 ... yes
25/62 ... yes
25/63 ... yes
12/65 ... yes
13/66 ... yes
13/67 ... yes
26/68 ... yes
13/69 ... yes
42/70 ... yes
13/71 ... yes
13/72 ... yes
36/73 ... yes
36/74 ... yes
41/75 ... yes
42/76 ... yes
14/77 ... yes
24/79 ... yes
45/80 ... yes
42/82 ... yes
12/83 ... yes
47/84 ... yes
42/85 ... yes
13/86 ... yes
13/87 ... yes
50/89 ... yes
14/91 ... yes
50/93 ... yes
65/94 ... yes
7/95 ... yes
14/96 ... yes
47/97 ... yes
12/98 ... yes
14/99 ... yes
13/100 ... yes
69/101 ... yes
14/102 ... yes
14/103 ... yes
65/104 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.8.4)
Active users: 38

Checking GitLab ... Finished

Results of GitLab environment info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.3.3p222
Gem Version:    2.6.6
Bundler Version:1.13.6
Rake Version:   10.5.0
Sidekiq Version:4.2.7

GitLab information
Version:        8.15.1
Revision:       1119c1f
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://git.fivenine.ch
HTTP Clone URL: https://git.fivenine.ch/some-group/some-project.git
SSH Clone URL:  git@git.fivenine.ch:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: github, twitter, google_oauth2

GitLab Shell
Version:        4.1.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git

Possible fixes

I don't know where the issue is exactly, but as a temporary workaround I change the role of the user from Developer to something else and back, then it works with cloning for a few hours.