Reconfiguration without relative urls does not remove 'relative_url.rb'
Summary
We used to access GitLab via a relative URL (http://xx.xx.xx.xx/git) and recently we decided to establish a dedicated domain (https://git.xxx.org).
According to the documentation, we first enable SSL on a non-bundled NGINX, and then configure it to connect GitLab via unix socket (with X-Forwarded-Ssl = on; X-Forwarded-Proto = $scheme configured). However, we can't login into GitLab any more, and it shows 422 "The change you requested was rejected on login".
We have looked into production.log
and discovered something interesting:
-
One requested asset still uses the relative URL:
Started GET "/git/assets/webpack/emoji.5dd6c6742df63f581cb0.chunk.js" for xx.xx.xx.xx at 2017-08-13 17:26:25 +0800 Processing by ApplicationController#route_not_found as JS Parameters: {"unmatched_route"=>"git/assets/webpack/emoji.5dd6c6742df63f581cb0.chunk"} Completed 401 Unauthorized in 5ms (ActiveRecord: 0.0ms)
-
The authenticity of CSRF token is failed to verify
Started POST "/users/sign_in" for xx.xx.xx.xx at 2017-08-13 17:53:54 +0800 Processing by SessionsController#create as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"MFjED6iOrRwiun6SIBslN2q/qYJ1AhOecFxwhrhHQiJ/L62uyTW5kE2UvFo2fOGDt8p+l70IejkxIwg==", "user"=>{"login"=>"[[USERNAME]]", "password"=>"[FILTERED]", "remember_me"=>"0"}} Can't verify CSRF token authenticity Completed 422 Unprocessable Entity in 25ms (ActiveRecord: 2.1ms) ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken): lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in `call' lib/gitlab/middleware/multipart.rb:93:in `call' lib/gitlab/request_profiler/middleware.rb:14:in `call' lib/gitlab/middleware/go.rb:16:in `call' lib/gitlab/etag_caching/middleware.rb:11:in `call' lib/gitlab/request_context.rb:18:in `call' lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
The second problem persists even after we reverted back to non-SSL and old relative URL settings.
Steps to reproduce
- Go to homepage
- Type user name and password
- Click login
Example Project
Latest GitLab 9.4.4
What is the current bug behavior?
422 "The change you requested was rejected on login"
What is the expected correct behavior?
The user is logged in
Relevant logs and/or screenshots
Expand for production.log
Started GET "/" for xx.xx.xx.xx at 2017-08-13 17:53:51 +0800 Processing by RootController#index as HTML Redirected to http://git.xxx.org/users/sign_in Filter chain halted as :redirect_unlogged_user rendered or redirected Completed 302 Found in 8ms (ActiveRecord: 0.0ms) Started GET "/users/sign_in" for xx.xx.xx.xx at 2017-08-13 17:53:52 +0800 Processing by SessionsController#new as HTML Completed 200 OK in 158ms (Views: 101.7ms | ActiveRecord: 6.9ms) Started GET "/git/assets/webpack/emoji.5dd6c6742df63f581cb0.chunk.js" for xx.xx.xx.xx at 2017-08-13 17:53:52 +0800 Processing by ApplicationController#route_not_found as JS Parameters: {"unmatched_route"=>"git/assets/webpack/emoji.5dd6c6742df63f581cb0.chunk"} Completed 401 Unauthorized in 2ms (ActiveRecord: 0.0ms) Started POST "/users/sign_in" for xx.xx.xx.xx at 2017-08-13 17:53:54 +0800 Processing by SessionsController#create as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"MFjED6iOrRwiun6SIBslN2q/qYJ1AhOecFxwhrhHQiJ/L62uyTW5kE2UvFo2fOGDt8p+l70IejkxIwg==", "user"=>{"login"=>"[USER NAME]", "password"=>"[FILTERED]", "remember_me"=>"0"}} Can't verify CSRF token authenticity Completed 422 Unprocessable Entity in 25ms (ActiveRecord: 2.1ms)
ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken): lib/gitlab/performance_bar/peek_performance_bar_with_rack_body.rb:16:in
call' lib/gitlab/middleware/multipart.rb:93:in
call' lib/gitlab/request_profiler/middleware.rb:14:incall' lib/gitlab/middleware/go.rb:16:in
call' lib/gitlab/etag_caching/middleware.rb:11:incall' lib/gitlab/request_context.rb:18:in
call' lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
Output of checks
Results of GitLab environment info
Current configuration is non-SSL and a relative URL at "/git".
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.13.4 Sidekiq Version:5.0.0 Go Version: go1.6.3 linux/amd64
GitLab information Version: 9.4.4 Revision: 2c79d92 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://git.xxx.org/git HTTP Clone URL: http://git.xxx.org/git/some-group/some-project.git SSH Clone URL: git@git.xxx.org:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.3.1 Repository storage paths:
- default: /home/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Expand for NGINX configuration
GitLab
Modified from nginx http version
http://blog.phusion.nl/2012/04/21/tutorial-setting-up-gitlab-on-debian-6/
Modified fromhttps://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
Modified fromLines starting with two hashes (##) are comments with information.
Lines starting with one hash (#) are configuration parameters that can be uncommented.
##################################
CONTRIBUTING
##################################
If you change this file in a Merge Request, please also create
https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests
a Merge Request on###################################
configuration
###################################
See installation.md#using-https for additional HTTPS configuration details.
upstream gitlab-workhorse { server unix:/var/opt/gitlab/gitlab-workhorse/socket fail_timeout=0; }
map $http_upgrade $connection_upgrade_gitlab_ssl { default upgrade; '' close; }
NGINX 'combined' log format with filtered query strings
log_format gitlab_ssl_access $remote_addr - $remote_user [$time_local] "$request_method $gitlab_ssl_filtered_request_uri $server_protocol" $status $body_bytes_sent "$gitlab_ssl_filtered_http_referer" "$http_user_agent";
Remove private_token from the request URI
In: /foo?private_token=unfiltered&authenticity_token=unfiltered&rss_token=unfiltered&...
Out: /foo?private_token=[FILTERED]&authenticity_token=unfiltered&rss_token=unfiltered&...
map $request_uri $gitlab_ssl_temp_request_uri_1 { default $request_uri; ~(?i)^(?.)(?[?&]private[-_]token)=[^&](?.*)$ "$start$temp=[FILTERED]$rest"; }
Remove authenticity_token from the request URI
In: /foo?private_token=[FILTERED]&authenticity_token=unfiltered&rss_token=unfiltered&...
Out: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&rss_token=unfiltered&...
map $gitlab_ssl_temp_request_uri_1 $gitlab_ssl_temp_request_uri_2 { default $gitlab_ssl_temp_request_uri_1; ~(?i)^(?.)(?[?&]authenticity[-_]token)=[^&](?.*)$ "$start$temp=[FILTERED]$rest"; }
Remove rss_token from the request URI
In: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&rss_token=unfiltered&...
Out: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&rss_token=[FILTERED]&...
map $gitlab_ssl_temp_request_uri_2 $gitlab_ssl_filtered_request_uri { default $gitlab_ssl_temp_request_uri_2; ~(?i)^(?.)(?[?&]rss[-_]token)=[^&](?.*)$ "$start$temp=[FILTERED]$rest"; }
A version of the referer without the query string
map $http_referer $gitlab_ssl_filtered_http_referer { default $http_referer; ~^(?.*)? $temp; }
HTTPS host
server { listen 0.0.0.0:80; listen [::]:80; server_name git.xxx.org; ## Replace this with something like gitlab.example.com server_tokens off; ## Don't show the nginx version number, a security best practice
Strong SSL Security
https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
#ssl on; #ssl_certificate /etc/dehydrated/certs/git.xxx.org/fullchain.pem; #ssl_certificate_key /etc/dehydrated/certs/git.xxx.org/privkey.pem;
GitLab needs backwards compatible ciphers to retain compatibioity with Java IDEs
#ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"; #ssl_protocols TLSv1 TLSv1.1 TLSv1.2; #ssl_prefer_server_ciphers on; #ssl_session_cache shared:SSL:10m; #ssl_session_timeout 5m;
See app/controllers/application_controller.rb for headers set
[Optional] If your certficate has OCSP, enable OCSP stapling to reduce the overhead and latency of running SSL.
Replace with your ssl_trusted_certificate. For more info see:
https://medium.com/devops-programming/4445f4862461
-https://www.ruby-forum.com/topic/4419319
-https://www.digitalocean.com/community/tutorials/how-to-configure-ocsp-stapling-on-apache-and-nginx
-ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/nginx/ssl/stapling.trusted.crt;
resolver 208.67.222.222 208.67.222.220 valid=300s; # Can change to your DNS resolver if desired
resolver_timeout 5s;
[Optional] Generate a stronger DHE parameter:
sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096
ssl_dhparam /etc/ssl/certs/dhparam.pem;
[Optional] Enable HTTP Strict Transport Security
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
Real IP Module Config
http://nginx.org/en/docs/http/ngx_http_realip_module.html
real_ip_header X-Real-IP; ## X-Real-IP or X-Forwarded-For or proxy_protocol real_ip_recursive off; ## If you enable 'on'
If you have a trusted IP address, uncomment it and set it
set_real_ip_from YOUR_TRUSTED_ADDRESS; ## Replace this with something like 192.168.1.0/24
Individual nginx logs for this GitLab vhost
access_log /var/log/nginx/gitlab_access.log gitlab_ssl_access; error_log /var/log/nginx/gitlab_error.log;
location ^~ /git { client_max_body_size 0; gzip off;
## https://github.com/gitlabhq/gitlabhq/issues/694 ## Some requests take more than 30 seconds. proxy_read_timeout 300; proxy_connect_timeout 300; proxy_redirect off; proxy_http_version 1.1; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Upgrade $http_upgrade; # proxy_set_header X-Forwarded-Ssl on; # proxy_set_header Connection $connection_upgrade_gitlab_ssl; proxy_pass http://gitlab-workhorse;
}
error_page 404 /404.html; error_page 422 /422.html; error_page 500 /500.html; error_page 502 /502.html; error_page 503 /503.html; location ~ ^/(404|422|500|502|503).html$ { root /opt/gitlab/embedded/service/gitlab-rails/public; internal; } }
Expand for GitLab configuration
external_url 'http://git.xxx.org/git'
gitlab_rails['time_zone'] = 'Asia/Shanghai' gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_email_from'] = 'xxx@gmail.com' gitlab_rails['gitlab_email_display_name'] = 'xxx' gitlab_rails['gitlab_email_reply_to'] = 'noreply@xxx.org'
gitlab_rails['lfs_enabled'] = true gitlab_rails['lfs_storage_path'] = "/home/git-lfs"
gitlab_rails['backup_upload_connection'] = { :provider => 'Local', :local_root => '/mnt/backup' } gitlab_rails['backup_upload_remote_directory'] = 'gitlab-tmp'
git_data_dir "/home/git-data"
web_server['external_users'] = ['root', 'nginx']
nginx['enable'] = false nginx['listen_addresses'] = ['0.0.0.0'] nginx['listen_port'] = 6600 # override only if you use a reverse proxy: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#setting-the-nginx-listen-port nginx['listen_https'] = false # override only if your reverse proxy internally communicates over HTTP: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#supporting-proxied-ssl nginx['proxy_set_headers'] = { "X-Forwarded-Proto" => "https", "X-Forwarded-Ssl" => "on" }
Results of GitLab application Check
Expand for gitlab:check output
Checking GitLab Shell ...
GitLab Shell version >= 5.3.1 ? ... OK (5.3.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/1 ... ok 2/2 ... ok 2/4 ... ok 5/6 ... ok 4/7 ... ok 5/8 ... ok 3/9 ... ok 10/10 ... repository is empty 3/11 ... ok 12/12 ... ok 9/20 ... ok 9/21 ... ok 3/22 ... ok 16/23 ... ok 5/24 ... ok 18/25 ... ok 15/26 ... ok 7/27 ... ok 3/28 ... ok 4/29 ... ok 16/30 ... ok 12/31 ... ok 3/32 ... ok 4/33 ... ok 5/35 ... ok 8/36 ... ok 23/38 ... ok 23/39 ... ok 28/42 ... ok 3/43 ... ok 28/46 ... ok 28/48 ... ok 20/49 ... ok 4/50 ... ok 20/51 ... ok 15/53 ... ok 20/54 ... ok 33/56 ... ok 20/57 ... ok 30/58 ... ok 22/60 ... ok 12/61 ... ok 5/63 ... ok 20/64 ... ok 33/65 ... ok 4/66 ... ok 33/67 ... ok 5/68 ... ok 5/69 ... ok 5/70 ... ok 33/71 ... ok 28/72 ... ok 5/73 ... ok 28/74 ... ok 30/75 ... ok 30/76 ... repository is empty 7/77 ... ok 7/78 ... ok 33/79 ... ok 8/80 ... ok 33/81 ... ok 33/82 ... ok 33/83 ... ok 33/84 ... ok 33/85 ... ok 20/86 ... ok 38/87 ... ok 28/88 ... ok 3/89 ... ok 4/90 ... ok 47/91 ... ok 20/92 ... repository is empty 4/93 ... ok 5/94 ... ok 33/95 ... ok 43/96 ... repository is empty 3/97 ... ok 28/98 ... ok 7/99 ... ok 28/100 ... repository is empty 8/101 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes 2/4 ... yes 5/6 ... yes 4/7 ... yes 5/8 ... yes 3/9 ... yes 10/10 ... yes 3/11 ... yes 12/12 ... yes 9/20 ... yes 9/21 ... yes 3/22 ... yes 16/23 ... yes 5/24 ... yes 18/25 ... yes 15/26 ... yes 7/27 ... yes 3/28 ... yes 4/29 ... yes 16/30 ... yes 12/31 ... yes 3/32 ... yes 4/33 ... yes 5/35 ... yes 8/36 ... yes 23/38 ... yes 23/39 ... yes 28/42 ... yes 3/43 ... yes 28/46 ... yes 28/48 ... yes 20/49 ... yes 4/50 ... yes 20/51 ... yes 15/53 ... yes 20/54 ... yes 33/56 ... yes 20/57 ... yes 30/58 ... yes 22/60 ... yes 12/61 ... yes 5/63 ... yes 20/64 ... yes 33/65 ... yes 4/66 ... yes 33/67 ... yes 5/68 ... yes 5/69 ... yes 5/70 ... yes 33/71 ... yes 28/72 ... yes 5/73 ... yes 28/74 ... yes 30/75 ... yes 30/76 ... yes 7/77 ... yes 7/78 ... yes 33/79 ... yes 8/80 ... yes 33/81 ... yes 33/82 ... yes 33/83 ... yes 33/84 ... yes 33/85 ... yes 20/86 ... yes 38/87 ... yes 28/88 ... yes 3/89 ... yes 4/90 ... yes 47/91 ... yes 20/92 ... yes 4/93 ... yes 5/94 ... yes 33/95 ... yes 43/96 ... yes 3/97 ... yes 28/98 ... yes 7/99 ... yes 28/100 ... yes 8/101 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.4) Active users: ... 44
Checking GitLab ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)