GitLab runner: repository not found

Summary

At random intervals, my gitlab-runner fails to fetch or clone the repo. If I wait a few minutes and retry the build, it starts working again, then fails a few builds later. This can happen both at the build or deploy stage.

Steps to reproduce

On an omnibus install that otherwise works fine, configure a gitlab runner and wait for triggered builds to randomly fail.

The runner is on the same OS as the server, and both are running on a VM that is networked behind a reverse-proxy. The URL for the repo refers to the external address of the server, so requests from the runner to the repo go through the rev-proxy, but most of the time this works fine.

The message when a failure happens is:

Running with gitlab-ci-multi-runner 1.7.1 (f896af7)
Using Shell executor...
Running on sirup...
Fetching changes...
HEAD is now at 4f51e09 Updated and slimmed down exerciser
fatal: repository 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.siruplab.com/lelivrescolaire/lls-tablet-android.git/' not found
ERROR: Build failed: exit status 1

I thought this might due to clock drift, but even after a manual ntpdate resync, it still happens. Besides, the runner is on the same system as the server.

Expected behavior

The runner should work consistently.

Actual behavior

It doesn't…

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Output of checks

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 3.6.6 ? ... OK (3.6.6)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/3 ... ok
4/4 ... ok
5/5 ... ok
5/6 ... ok
5/7 ... ok
5/8 ... ok
7/9 ... ok
4/10 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
4/3 ... yes
4/4 ... yes
5/5 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
7/9 ... yes
4/10 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 7

Checking GitLab ... Finished

Results of GitLab environment info

System information
System:         Debian 8.6
Current User:   git
Using RVM:      no
Ruby Version:   2.3.1p112
Gem Version:    2.6.6
Bundler Version:1.13.6
Rake Version:   10.5.0
Sidekiq Version:4.2.1

GitLab information
Version:        8.13.5
Revision:       09cedb5
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.siruplab.com
HTTP Clone URL: https://gitlab.siruplab.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.siruplab.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: gitlab, github

GitLab Shell
Version:        3.6.6
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git