Random 403 behind nginx proxy
Summary
I run a Gitlab CE server behind Nginx proxy, and regularly after 3 or 4 builds, the server would stop accepting me by throwing 403 to every request I do. If I try to connect with any other user it would work, but even as an admin user, if I impersonate my user it would give me white page. After a day or two, things would go back to normal.
Steps to reproduce
Gitlab CE (up to date) behind nginx (1.13.12) over HTTP/2
Run multiple pipelines (with some of them failing) within a few hours
Maybe other steps but I'm not sure as the problem is not really straightforward
Example Project
Not possible as I never had the problem on gitlab.com on the same projects
What is the current bug behavior?
White page (underlying 403) for all pages as long as I'm connected (if I delete the session cookie, I can access the website until I connect again)
What is the expected correct behavior?
The actual Gitlab service
Relevant logs and/or screenshots
production.log
Started GET "/" for <my-ip> at 2018-04-12 16:21:53 +0200
Processing by RootController#index as HTML
Completed 403 Forbidden in 5ms (ActiveRecord: 0.9ms)
production_json.log
{"method":"GET","path":"/","format":"html","controller":"RootController","action":"index","status":403,"duration":5.89,"view":0.0,"db":0.89,"time":"2018-04-12T14:21:53.535Z","params":[],"remote_ip":"<my-ip>","user_id":3,"username":"grobert"}
nginx access.log
<my-ip> - - [12/Apr/2018:16:21:53 +0200] "GET / HTTP/2.0" 403 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0"
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 8 Current User: gitlab Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.6.4 Revision: dee2c87 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.minesouls.fr HTTP Clone URL: https://gitlab.minesouls.fr/some-group/some-project.git SSH Clone URL: gitlab@ssh.minesouls.fr:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 6.0.4 Repository storage paths:
- default: /home/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 6.0.4 ? ... OK (6.0.4) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by gitlab:root, or gitlab:gitlab? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 4/3 ... ok 5/6 ... ok 4/9 ... ok 4/10 ... ok 4/11 ... ok 4/12 ... ok 3/13 ... ok 3/15 ... ok 3/16 ... ok 3/17 ... ok 2/18 ... ok 2/19 ... ok 3/20 ... ok 3/21 ... ok 2/22 ... ok 2/23 ... ok 15/25 ... ok 16/26 ... ok 18/27 ... ok 19/28 ... ok 20/29 ... ok 20/30 ... ok 3/31 ... ok 15/33 ... ok 15/34 ... ok 19/35 ... ok 26/36 ... ok 26/37 ... ok 26/38 ... ok 26/39 ... ok 27/41 ... ok 27/42 ... ok 3/43 ... ok 3/44 ... ok 3/45 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/3 ... yes 5/6 ... yes 4/9 ... yes 4/10 ... yes 4/11 ... yes 4/12 ... yes 3/13 ... yes 3/15 ... yes 3/16 ... yes 3/17 ... yes 2/18 ... yes 2/19 ... yes 3/20 ... yes 3/21 ... yes 2/22 ... yes 2/23 ... yes 15/25 ... yes 16/26 ... yes 18/27 ... yes 19/28 ... yes 20/29 ... yes 20/30 ... yes 3/31 ... yes 15/33 ... yes 15/34 ... yes 19/35 ... yes 26/36 ... yes 26/37 ... yes 26/38 ... yes 26/39 ... yes 27/41 ... yes 27/42 ... yes 3/43 ... yes 3/44 ... yes 3/45 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 12
Checking GitLab ... Finished
Feel free to ask if you need any more information