GitLab behind nginx proxy-pass and CloudFront returns 404 on signatures file

Summary

When viewing the commit log for a repository on my personal GitLab, it displays a notice "Something went wrong on our end." Viewing the Developer Tools in my browser, I see that the following file is returning 404:

https://gitlab.kode54.net/kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search=

Steps to reproduce

  1. Install GitLab.
  2. Configure it so nginx listens on 127.0.0.1.
  3. Configure nginx with custom configure path.
  4. Configure custom file to proxy_pass the location / to 127.0.0.1:443, with the correct Host header for the following.
  5. Configure an Amazon CloudFront distribution to source from the server, using the public facing hostname as the CNAME.
  6. Open a project in the public GitLab.
  7. Click on the Commits log link.

Example Project

https://gitlab.kode54.net/kode54/deadbeef/

What is the current bug behavior?

The signatures file returns a 404 error.

What is the expected correct behavior?

The signatures file returns whatever it's supposed to, so the commit log page doesn't display an error.

Relevant logs and/or screenshots

==> /var/log/gitlab/nginx/access.log <==

35.167.191.163 - - [10/Oct/2017:16:32:52 -0700] "GET /kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search= HTTP/1.1" 404 1565 "-" "Amazon CloudFront"

==> /var/log/gitlab/nginx/gitlab_access.log <==

127.0.0.1 - - [10/Oct/2017:16:32:52 -0700] "GET /kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search= HTTP/1.0" 404 3084 "" "Amazon CloudFront"

==> /var/log/gitlab/gitlab-workhorse/current <==

2017-10-10_23:32:52.07646 gitlab.kode54.net @ - - [2017-10-10 16:32:51.918129668 -0700 PDT] "GET /kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search= HTTP/1.1" 404 3084 "" "Amazon CloudFront" 0.158254

==> /var/log/gitlab/gitlab-rails/production.log <==

Started GET "/kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search=" for x.x.x.x at 2017-10-10 16:39:42 -0700
Processing by Projects::CommitsController#signatures as JS
  Parameters: {"utf8"=>"✓", "search"=>"", "namespace_id"=>"kode54", "project_id"=>"deadbeef", "id"=>"master"}
Started GET "/kode54/deadbeef/commits/master" for x.x.x.x at 2017-10-10 16:39:43 -0700
Processing by Projects::CommitsController#show as HTML
  Parameters: {"namespace_id"=>"kode54", "project_id"=>"deadbeef", "id"=>"master"}
Completed 404 Not Found in 195ms (ActiveRecord: 10.6ms)

==> /var/log/gitlab/gitlab-rails/production_json.log <==

{"method":"GET","path":"/kode54/deadbeef/commits/master/signatures","format":"js","controller":"Projects::CommitsController","action":"signatures","status":404,"duration":195.51,"view":0.0,"db":10.58,"time":"2017-10-10T23:39:43.002Z","params":{"utf8":"✓","search":"","namespace_id":"kode54","project_id":"deadbeef","id":"master"},"remote_ip":"x.x.x.x","user_id":2,"username":"kode54"}

==> /var/log/gitlab/gitlab-workhorse/current <==

2017-10-10_23:39:43.19880 gitlab.kode54.net @ - - [2017-10-10 16:39:42.99474243 -0700 PDT] "GET /kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search= HTTP/1.1" 404 3084 "" "Amazon CloudFront" 0.203977

==> /var/log/gitlab/nginx/gitlab_access.log <==

127.0.0.1 - - [10/Oct/2017:16:39:43 -0700] "GET /kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search= HTTP/1.0" 404 3084 "" "Amazon CloudFront"

==> /var/log/gitlab/nginx/access.log <==

35.167.191.163 - - [10/Oct/2017:16:39:43 -0700] "GET /kode54/deadbeef/commits/master/signatures?utf8=%E2%9C%93&search= HTTP/1.1" 404 1565 "-" "Amazon CloudFront"

Output of checks

Results of GitLab environment info


System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.5p376
Gem Version:	2.6.13
Bundler Version:1.13.7
Rake Version:	12.0.0
Redis Version:	3.2.5
Git Version:	2.13.5
Sidekiq Version:5.0.4
Go Version:	unknown

GitLab information
Version:	10.0.3
Revision:	8895150
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://gitlab.kode54.net
HTTP Clone URL:	https://gitlab.kode54.net/some-group/some-project.git
SSH Clone URL:	git@home.kode54.net:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: github, bitbucket

GitLab Shell
Version:	5.9.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 5.9.0 ? ... OK (5.9.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
2/1 ... ok
2/2 ... ok
2/3 ... ok
2/4 ... ok
2/5 ... ok
2/6 ... ok
2/7 ... ok
2/8 ... ok
2/9 ... ok
2/10 ... ok
2/11 ... ok
2/12 ... ok
2/13 ... ok
2/14 ... ok
2/15 ... ok
2/16 ... ok
2/17 ... ok
2/18 ... ok
2/19 ... ok
2/20 ... ok
2/21 ... ok
2/22 ... ok
2/23 ... ok
2/24 ... ok
2/25 ... ok
2/26 ... ok
2/27 ... ok
2/28 ... ok
2/29 ... ok
2/30 ... ok
2/31 ... ok
2/33 ... ok
2/34 ... ok
2/35 ... ok
2/36 ... ok
2/37 ... ok
2/38 ... ok
2/39 ... ok
2/40 ... ok
2/41 ... ok
2/42 ... ok
2/43 ... ok
2/44 ... ok
2/45 ... ok
2/46 ... ok
2/47 ... ok
2/48 ... ok
2/49 ... ok
2/50 ... ok
2/51 ... ok
2/52 ... ok
2/53 ... ok
2/54 ... ok
2/55 ... ok
2/56 ... ok
2/57 ... ok
2/58 ... ok
2/59 ... ok
2/60 ... ok
2/61 ... ok
2/62 ... ok
2/63 ... ok
2/64 ... ok
2/65 ... ok
2/66 ... ok
2/67 ... ok
2/68 ... ok
2/69 ... ok
2/70 ... ok
2/71 ... ok
2/72 ... ok
2/73 ... ok
2/74 ... ok
2/75 ... ok
2/76 ... ok
2/77 ... ok
2/78 ... ok
2/79 ... ok
2/80 ... ok
2/81 ... ok
2/82 ... ok
2/84 ... ok
6/85 ... ok
2/86 ... ok
2/87 ... ok
2/88 ... ok
2/90 ... ok
2/92 ... ok
2/94 ... ok
16/95 ... ok
2/96 ... ok
2/97 ... ok
2/98 ... ok
2/99 ... ok
43/100 ... ok
2/101 ... ok
34/104 ... ok
2/105 ... ok
50/106 ... ok
43/107 ... ok
2/109 ... ok
34/111 ... ok
34/112 ... ok
34/113 ... ok
34/114 ... ok
34/116 ... ok
34/117 ... ok
34/118 ... ok
34/120 ... ok
44/121 ... ok
2/122 ... ok
34/123 ... ok
34/124 ... ok
44/125 ... ok
34/126 ... repository is empty
2/127 ... ok
2/128 ... ok
34/129 ... ok
2/130 ... ok
34/131 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: FAILED. code: 403
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /opt/gitlab/embedded/service/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml
Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
2/1 ... yes
2/2 ... yes
2/3 ... yes
2/4 ... yes
2/5 ... yes
2/6 ... yes
2/7 ... yes
2/8 ... yes
2/9 ... yes
2/10 ... yes
2/11 ... yes
2/12 ... yes
2/13 ... yes
2/14 ... yes
2/15 ... yes
2/16 ... yes
2/17 ... yes
2/18 ... yes
2/19 ... yes
2/20 ... yes
2/21 ... yes
2/22 ... yes
2/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
2/27 ... yes
2/28 ... yes
2/29 ... yes
2/30 ... yes
2/31 ... yes
2/33 ... yes
2/34 ... yes
2/35 ... yes
2/36 ... yes
2/37 ... yes
2/38 ... yes
2/39 ... yes
2/40 ... yes
2/41 ... yes
2/42 ... yes
2/43 ... yes
2/44 ... yes
2/45 ... yes
2/46 ... yes
2/47 ... yes
2/48 ... yes
2/49 ... yes
2/50 ... yes
2/51 ... yes
2/52 ... yes
2/53 ... yes
2/54 ... yes
2/55 ... yes
2/56 ... yes
2/57 ... yes
2/58 ... yes
2/59 ... yes
2/60 ... yes
2/61 ... yes
2/62 ... yes
2/63 ... yes
2/64 ... yes
2/65 ... yes
2/66 ... yes
2/67 ... yes
2/68 ... yes
2/69 ... yes
2/70 ... yes
2/71 ... yes
2/72 ... yes
2/73 ... yes
2/74 ... yes
2/75 ... yes
2/76 ... yes
2/77 ... yes
2/78 ... yes
2/79 ... yes
2/80 ... yes
2/81 ... yes
2/82 ... yes
2/84 ... yes
6/85 ... yes
2/86 ... yes
2/87 ... yes
2/88 ... yes
2/90 ... yes
2/92 ... yes
2/94 ... yes
16/95 ... yes
2/96 ... yes
2/97 ... yes
2/98 ... yes
2/99 ... yes
43/100 ... yes
2/101 ... yes
34/104 ... yes
2/105 ... yes
50/106 ... yes
43/107 ... yes
2/109 ... yes
34/111 ... yes
34/112 ... yes
34/113 ... yes
34/114 ... yes
34/116 ... yes
34/117 ... yes
34/118 ... yes
34/120 ... yes
44/121 ... yes
2/122 ... yes
34/123 ... yes
34/124 ... yes
44/125 ... yes
34/126 ... yes
2/127 ... yes
2/128 ... yes
34/129 ... yes
2/130 ... yes
34/131 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.13.5)
Git user has default SSH configuration? ... yes
Active users: ... 150

Checking GitLab ... Finished

(we will only investigate if the tests are passing)

How nice, your tests are failing because your API self-check is hitting a 403 forbidden error, for reasons beyond my control. I can access:

https://gitlab.kode54.net/api/v4/internal/check

And it returns a JSON 401 Unauthorized error.

Please tell me this API URL is not expected to be reaching localhost from within the check machine, since it's roundtripping out through CloudFront and back through my router again before hitting the test server.

Possible fixes

I haven't a clue how to fix this, other than, obviously, paying $20/mo bare minimum for a large enough VPS to host this thing outside of my home again.

Edited by Christopher Snowhill