Cloning repo via gitlab-ci gives 403 in 9.0.2
Summary
After updating our Gitlab (omnibus) to 9.0.2, we ran into an error where running pipelines gives the following:
Cloning repository...
Cloning into '/builds/<group>/<repo>'...
remote: Forbidden
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.our-gitlab-server.com/<group>/<repo>.git/': The requested URL returned error: 403
ERROR: Job failed: exit code 1````
I did verify that clone via HTTPS is enabled (ssh and https both turned on).
### Steps to reproduce
This happens every time I fire off a CI job whether from a push or manually creating the pipeline via the web interface.
### What is the current *bug* behavior?
A 403 error is thrown.
### What is the expected *correct* behavior?
The pipeline should finish checkout.
### Relevant logs and/or screenshots
None that I could find - let me know where I can pull from to show this problem.
### Output of checks
````Checking GitLab Shell ...
GitLab Shell version >= 5.0.0 ? ... OK (5.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
1/1 ... ok
2/3 ... ok
2/4 ... ok
2/5 ... ok
2/6 ... ok
2/8 ... ok
11/9 ... ok
3/10 ... ok
12/12 ... ok
13/14 ... ok
12/15 ... ok
14/16 ... ok
2/17 ... ok
15/18 ... ok
15/19 ... ok
14/20 ... ok
12/21 ... ok
16/24 ... ok
16/25 ... ok
2/26 ... ok
2/27 ... ok
2/28 ... ok
3/29 ... ok
15/30 ... ok
18/31 ... ok
15/36 ... ok
19/37 ... ok
19/38 ... ok
3/39 ... ok
2/40 ... ok
12/41 ... ok
2/42 ... ok
19/43 ... ok
19/44 ... ok
2/45 ... ok
2/46 ... ok
2/47 ... ok
2/48 ... ok
3/49 ... ok
21/50 ... ok
3/51 ... ok
19/52 ... ok
23/54 ... ok
24/55 ... ok
1/56 ... ok
25/57 ... ok
23/58 ... ok
2/59 ... ok
3/60 ... ok
20/61 ... ok
19/62 ... ok
24/63 ... ok
24/64 ... ok
24/65 ... ok
33/66 ... ok
2/67 ... ok
15/68 ... ok
34/69 ... ok
35/70 ... ok
2/71 ... ok
35/74 ... ok
2/75 ... ok
19/76 ... ok
2/77 ... ok
12/78 ... ok
39/79 ... ok
2/80 ... ok
2/81 ... ok
2/82 ... ok
2/83 ... ok
44/84 ... ok
35/85 ... ok
2/86 ... ok
2/87 ... ok
19/88 ... ok
47/90 ... ok
35/91 ... ok
2/92 ... ok
55/93 ... ok
55/94 ... ok
55/95 ... ok
12/96 ... ok
12/97 ... ok
12/98 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
1/1 ... yes
2/3 ... yes
2/4 ... yes
2/5 ... yes
2/6 ... yes
2/8 ... yes
11/9 ... yes
3/10 ... yes
12/12 ... yes
13/14 ... yes
12/15 ... yes
14/16 ... yes
2/17 ... yes
15/18 ... yes
15/19 ... yes
14/20 ... yes
12/21 ... yes
16/24 ... yes
16/25 ... yes
2/26 ... yes
2/27 ... yes
2/28 ... yes
3/29 ... yes
15/30 ... yes
18/31 ... yes
15/36 ... yes
19/37 ... yes
19/38 ... yes
3/39 ... yes
2/40 ... yes
12/41 ... yes
2/42 ... yes
19/43 ... yes
19/44 ... yes
2/45 ... yes
2/46 ... yes
2/47 ... yes
2/48 ... yes
3/49 ... yes
21/50 ... yes
3/51 ... yes
19/52 ... yes
23/54 ... yes
24/55 ... yes
1/56 ... yes
25/57 ... yes
23/58 ... yes
2/59 ... yes
3/60 ... yes
20/61 ... yes
19/62 ... yes
24/63 ... yes
24/64 ... yes
24/65 ... yes
33/66 ... yes
2/67 ... yes
15/68 ... yes
34/69 ... yes
35/70 ... yes
2/71 ... yes
35/74 ... yes
2/75 ... yes
19/76 ... yes
2/77 ... yes
12/78 ... yes
39/79 ... yes
2/80 ... yes
2/81 ... yes
2/82 ... yes
2/83 ... yes
44/84 ... yes
35/85 ... yes
2/86 ... yes
2/87 ... yes
19/88 ... yes
47/90 ... yes
35/91 ... yes
2/92 ... yes
55/93 ... yes
55/94 ... yes
55/95 ... yes
12/96 ... yes
12/97 ... yes
12/98 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 22
Checking GitLab ... Finished````
#### Results of GitLab environment info
Unfortunately I've already downgraded back to 9.0.1, but here's the output regardless with the URLs sanitized.
System information
System: Ubuntu 16.04
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.10.2
Sidekiq Version:4.2.7
GitLab information
Version: 9.0.1
Revision: 0fa55cc
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://gitlab.pas-digital.com
HTTP Clone URL: https://gitlab.our-domain.com/some-group/some-project.git
SSH Clone URL: git@gitlab.our-domain.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 5.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git