Unable to fetch token when enabling pages access control in combination with -inplace-chroot
Summary
After upgrading to GitLab CE 11.5, we tried to setup Access Control for GitLab Pages.
We are running GitLab Omnibus in its docker version behind an Apache reverse proxy.
Steps to reproduce
Our tests have been made on a project set with a Private visibility and Pages permissions set as Only for Project Members.
We are using custom certificates but they are not self-signed.
Here is our configuration for GitLab pages in gitlab.rb config file:
gitlab_pages['enable'] = true
##! Configure to use JSON structured logging in GitLab Pages
gitlab_pages['log_format'] = "json"
##! Listen for requests forwarded by reverse proxy
gitlab_pages['listen_proxy'] = "0.0.0.0:8090"
gitlab_pages['redirect_http'] = true
gitlab_pages['dir'] = "/var/opt/gitlab/gitlab-pages"
gitlab_pages['log_directory'] = "/var/log/gitlab/gitlab-pages"
gitlab_pages['cert'] = "/etc/gitlab/ssl/domain.pem"
gitlab_pages['cert_key'] = "/etc/gitlab/ssl/domain.key"
gitlab_pages['inplace_chroot'] = true
##! Pages access control
gitlab_pages['access_control'] = true
## GitLab Pages NGINX
pages_nginx['enable'] = false
We have tried to change many parameters such as gitlab_pages['redirect_http'] to false or using pages_nginx['enable']
I know that Apache proxy is not officially supported but if you want, here is our configuration file
<VirtualHost "*:80">
ServerName pages.domain.fr
ServerAlias *.pages.domain.fr
ErrorLog /var/log/apache2/gitlab-pages/error.log
CustomLog /var/log/apache2/gitlab-pages/access.log combined
AllowEncodedSlashes NoDecode
# Proxy healthcheck
ProxyHCExpr ok234 {%{REQUEST_STATUS} =~ /^[234]/}
<Proxy "balancer://pagesHA/">
BalancerMember http://host1:12890 connectiontimeout=1 hcmethod=HEAD hcexpr=ok234 hcinterval=3 hcpasses=5 hcfails=2
BalancerMember http://host2:12890 connectiontimeout=1 hcmethod=HEAD hcexpr=ok234 hcinterval=3 hcpasses=5 hcfails=2
ProxySet lbmethod=bytraffic
Require all granted
</Proxy>
ProxyPass / balancer://pagesHA/
ProxyPassReverse / balancer://pagesHA/
RewriteEngine On
RewriteCond %{HTTP_HOST} ^(.*)\.pages\.domain\.fr$
RewriteRule (.*) https://%{HTTP_POST}%{REQUEST_URI} [L]
</VirtualHost>
<IfModule mod_ssl.c>
<VirtualHost "*:443">
ServerName pages.domain.fr
ServerAlias *.pages.domain.fr
ErrorLog /var/log/apache2/gitlab-pages/error.log
CustomLog /var/log/apache2/gitlab-pages/access.log combined
SSLEngine on
RewriteEngine on
SSLProxyEngine on
ProxyPreserveHost on
SSLProxyCheckPeerCN off
RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Ssl on
ProxyTimeout 60
SSLCertificateFile /etc/apache2/ssl/domain.pem
SSLCertificateKeyFile /etc/apache2/ssl/domain.key
SSLCertificateChainFile /etc/apache2/ssl/chain.crt
SSLCACertificateFile /etc/apache2/ssl/ca.crt
SSLVerifyClient none
SSLVerifyDepth 10
AllowEncodedSlashes NoDecode
# Proxy healthcheck
ProxyHCExpr ok234 {%{REQUEST_STATUS} =~ /^[234]/}
<Proxy "balancer://pagesHASSL/">
BalancerMember https://host1:12890 connectiontimeout=1 hcmethod=HEAD hcexpr=ok234 hcinterval=3 hcpasses=5 hcfails=2
BalancerMember https://host2:12890 connectiontimeout=1 hcmethod=HEAD hcexpr=ok234 hcinterval=3 hcpasses=5 hcfails=2
ProxySet lbmethod=bytraffic
Require all granted
</Proxy>
ProxyPass / balancer://pagesHASSL/
ProxyPassReverse / balancer://pagesHASSL/
</VirtualHost>
</IfModule>Expected behaviour:
When the user goes to its Project > Pages settings and click on the URL he should be able to access its website. What I understand is that Gitlab makes a requests to generate a unique token to let the user access the resource.
What is the current bug behavior ?
When trying to access the page, the server returns a 503 error. If we look more specifically to the logs we can see a network error while trying to get the access token.
{"uri":"/auth","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0","written":2904}
{"host":"mywebsite.pages.domain.fr","level":"debug","msg":"No access token exists, redirecting user to OAuth2 login","path":"/favicon.ico","time":"2018-11-28T08:27:18Z"}
{"duration":0.000582523,"host":"mywebsite.pages.domain.fr","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referer":"","remoteAddr":"172.18.1.1:52900","status":302,"system":"http","time":"2018-11-28T08:27:18Z","uri":"/favicon.ico","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0","written":155}
{"host":"projects.pages.domain.fr","level":"debug","msg":"Authentication callback","path":"/auth?domain=http://mywebsite.pages.domain.fr\u0026state=Lb8Eczf0qI2hQkuw90y2Yg==","time":"2018-11-28T08:27:18Z"}
{"domain":"http://mywebsite.pages.domain.fr","host":"projects.pages.domain.fr","level":"debug","msg":"User is authenticating via domain","path":"/auth?domain=http://mywebsite.pages.domain.fr\u0026state=Lb8Eczf0qI2hQkuw90y2Yg==","time":"2018-11-28T08:27:18Z"}
{"duration":0.000489011,"host":"projects.pages.domain.fr","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referer":"","remoteAddr":"172.18.1.1:52888","status":302,"system":"http","time":"2018-11-28T08:27:18Z","uri":"/auth","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0","written":265}
{"host":"projects.pages.domain.fr","level":"debug","msg":"Authentication callback","path":"/auth?code=aa7c5a7cb234d41524101f951121db52d4cdabdd2f9a25b50c9d6edb03cdc118\u0026state=Lb8Eczf0qI2hQkuw90y2Yg%3D%3D","time":"2018-11-28T08:27:18Z"}
{"host":"projects.pages.domain.fr","level":"debug","msg":"Redirecting auth callback to custom domain","path":"/auth?code=aa7c5a7cb234d41524101f951121db52d4cdabdd2f9a25b50c9d6edb03cdc118\u0026state=Lb8Eczf0qI2hQkuw90y2Yg%3D%3D","time":"2018-11-28T08:27:18Z"}
{"duration":0.000402059,"host":"projects.pages.domain.fr","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referer":"","remoteAddr":"172.18.1.1:52888","status":302,"system":"http","time":"2018-11-28T08:27:18Z","uri":"/auth","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0","written":182}
{"host":"mywebsite.pages.domain.fr","level":"debug","msg":"Authentication callback","path":"/auth?code=aa7c5a7cb234d41524101f951121db52d4cdabdd2f9a25b50c9d6edb03cdc118\u0026state=Lb8Eczf0qI2hQkuw90y2Yg%3D%3D","time":"2018-11-28T08:27:18Z"}
{"error":"Post https://gitlab.domain.fr/oauth/token: dial tcp: lookup gitlab.domain.fr on [::1]:53: dial udp [::1]:53: connect: network is unreachable","host":"mywebsite.pages.domain.fr","level":"debug","msg":"Fetching access token failed","path":"/auth?code=aa7c5a7cb234d41524101f951121db52d4cdabdd2f9a25b50c9d6edb03cdc118\u0026state=Lb8Eczf0qI2hQkuw90y2Yg%3D%3D","time":"2018-11-28T08:27:18Z"}
{"duration":0.001513229,"host":"mywebsite.pages.domain.fr","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referer":"","remoteAddr":"172.18.1.1:52900","status":503,"system":"http","time":"2018-11-28T08:27:18Z","uri":"/auth","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0","written":2904}Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.4.5p335 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.1 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.1 Go Version: unknownGitLab information Version: 11.5.0 Revision: b7b1e8e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.domain.fr HTTP Clone URL: https://gitlab.domain.fr/some-group/some-project.git SSH Clone URL: git@gitlab.domain.fr:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: shibboleth
GitLab Shell Version: 8.4.1 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ... GitLab Shell version >= 8.4.1 ? ... OK (8.4.1) hooks directories in repos are links: ... (Skipped because it's too long) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Reply by email is disabled in config/gitlab.yml Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... (Skipped because it's too long) Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.5) Git version >= 2.9.5 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 1242 Checking GitLab ... Finished
Discussion
I don't really understand if its a bug from the container itself of if its a misconfiguration from the Apache proxy ... I have check the apache & nginx logs without finding anything relevant. I'll be glad to hear if you guys have heard of this kind of problem recently.
Edited by Jason Yavorsky