pages daemon fails DNS lookup
Summary
When enabling authentication for the pages daemon it tries to do a call back to the gitlab server and that fails when doing a DNS lookup to find the address of the gitlab server:
level=debug msg=“Fetching access token failed” error=“Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address” host=project.pages.example.com path="/auth?code=46c1f78&state=7uw%3D%3D"
The pages daemon works when removing the -daemon-* options:
-daemon-uid=998 -daemon-gid=998 -daemon-inplace-chroot=true
... and running this command line as git:
/opt/gitlab/embedded/bin/gitlab-pages -listen-proxy=localhost:8090 -pages-domain=pages.example.com -pages-root=/var/opt/gitlab/gitlab-rails/shared/pages -log-verbose -redirect-http=false -use-http2=true -artifacts-server=https://gitlab.example.com/api/v4 -artifacts-server-timeout=10 -auth-client-id=snip -auth-client-secret=snip -auth-redirect-uri=http://projects.pages.example.com/auth -auth-server=https://gitlab.example.com/ -auth-secret=snip -admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret -admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket
Steps to reproduce
- Set up a gitlab server container using gitlab/gitlab-ce:11.9.4-ce.0
- Put the gitlab container behind an external load balancer that terminates TLS and exposes https://gitlab.example.com
- Enable pages with authentication according to the documentation and have the external LB expose its port as http://pages.example.com
- Observe that it breaks.
Workaround
See #4243 (comment 418829477) for details
cp /etc/resolv.conf /var/opt/gitlab/gitlab-rails/shared/pages/etc/resolv.conf
cp -rv /etc/ssl /var/opt/gitlab/gitlab-rails/shared/pages/etc/
gitlab-ctl restart gitlab-pages
What is the current bug behavior?
- DNS lookup fails when making call back to the gitlab server for authentication
What is the expected correct behavior?
- DNS lookup and authentication should work.
Relevant logs
Relevant logs
2019-04-03_12:56:21.03194 time="2019-04-03T12:56:21Z" level=info msg="GitLab Pages Daemon" revision=869b94c version=1.5.0 2019-04-03_12:56:21.03196 time="2019-04-03T12:56:21Z" level=info msg="URL: https://gitlab.com/gitlab-org/gitlab-pages" 2019-04-03_12:56:21.03203 time="2019-04-03T12:56:21Z" level=debug msg="Start daemon with configuration" admin-https-cert= admin-https-key= admin-https-listener= admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket artifacts-server="https://gitlab.example.com/api/v4" artifacts-server-timeout=10 auth-client-id=$snip auth-client-secret=$snip auth-redirect-uri="http://projects.pages.example.com/auth" auth-secret=$snip auth-server="https://gitlab.example.com/" daemon-gid=998 daemon-inplace-chroot=true daemon-uid=998 default-config-filename=config disable-cross-origin-requests=false domain=pages.example.com listen-http= listen-https= listen-proxy="localhost:8090" log-format=text metrics-address= pages-domain=pages.example.com pages-root=/var/opt/gitlab/gitlab-rails/shared/pages pages-status= redirect-http=false root-cert= root-key= status_path= use-http-2=true 2019-04-03_12:56:21.03229 time="2019-04-03T12:56:21Z" level=debug msg="Set up proxy listener" listener="localhost:8090" 2019-04-03_12:56:21.03512 time="2019-04-03T12:56:21Z" level=debug msg="Set up admin unix socket" listener=/var/opt/gitlab/gitlab-pages/admin.socket 2019-04-03_12:56:21.03513 time="2019-04-03T12:56:21Z" level=info msg="running the daemon as unprivileged user" gid=998 in-place=true uid=998 2019-04-03_12:56:21.06698 time="2019-04-03T12:56:21Z" level=info msg="starting the daemon as unprivileged user" gid=998 uid=998 2019-04-03_12:56:21.07488 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=6.6999e-05 group=containers 2019-04-03_12:56:21.07491 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=2.5998e-05 group=gradle 2019-04-03_12:56:21.07492 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.209e-05 group=rm 2019-04-03_12:56:21.07494 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=1.0072e-05 group=@pages.tmp 2019-04-03_12:56:21.07495 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.4679e-05 group=statistics 2019-04-03_12:56:21.07505 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000112507 group=techsupport 2019-04-03_12:56:21.07530 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000341842 group=project 2019-04-03_12:56:21.07532 time="2019-04-03T12:56:21Z" level=debug msg="Configured domain" domain=project host=project.pages.example.com 2019-04-03_12:56:21.07533 time="2019-04-03T12:56:21Z" level=info msg="Updated all domains" count(domains)=1 duration=0.001172813 hash=d9836b4b6a5aeca375ed7c4718672245d57fb2a9fe1dd75cf9d9263ee503c0f93d517526366bd6a67180d6f2f1886ff6187f0fbca21119e87cca683f1e718d11 2019-04-03_13:10:42.72420 time="2019-04-03T13:10:42Z" level=debug msg="Authenticate request" host=project.pages.example.com path=/test/hest 2019-04-03_13:10:42.72421 time="2019-04-03T13:10:42Z" level=debug msg="No access token exists, redirecting user to OAuth2 login" host=project.pages.example.com path=/test/hest 2019-04-03_13:10:42.72434 project.pages.example.com 127.0.0.1:54244 - - [2019-04-03 13:10:42.723822327 +0000 UTC m=+861.664076193] "GET /test/hest HTTP/1.0" 302 127 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000491 2019-04-03_13:10:42.73268 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip" 2019-04-03_13:10:42.73271 time="2019-04-03T13:10:42Z" level=debug msg="User is authenticating via domain" domain="http://project.pages.example.com" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip" 2019-04-03_13:10:42.73276 projects.pages.example.com 127.0.0.1:54248 - - [2019-04-03 13:10:42.732566565 +0000 UTC m=+861.672820431] "GET /auth HTTP/1.0" 302 248 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000183 2019-04-03_13:10:42.82561 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?code=$snip%3D%3D" 2019-04-03_13:10:42.82562 time="2019-04-03T13:10:42Z" level=debug msg="Redirecting auth callback to custom domain" host=projects.pages.example.com path="/auth?code=$snip%3D%3D" 2019-04-03_13:10:42.82568 projects.pages.example.com 127.0.0.1:54264 - - [2019-04-03 13:10:42.825393741 +0000 UTC m=+861.765647616] "GET /auth HTTP/1.0" 302 163 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000260 2019-04-03_13:10:42.83338 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=project.pages.example.com path="/auth?code=$snip%3D%3D" 2019-04-03_13:10:42.83371 time="2019-04-03T13:10:42Z" level=error msg="failed to read SSL_CERT_FILE" error="open /opt/gitlab/embedded/ssl/certs/cacert.pem: no such file or directory" 2019-04-03_13:10:42.83435 time="2019-04-03T13:10:42Z" level=debug msg="Fetching access token failed" error="Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address" host=project.pages.example.com path="/auth?code=$snip%3D%3D" 2019-04-03_13:10:42.83442 project.pages.example.com 127.0.0.1:54268 - - [2019-04-03 13:10:42.832351673 +0000 UTC m=+861.772605539] "GET /auth HTTP/1.0" 503 2904 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.002045
Details of package version
Provide the package version installation details
gitlab-ce 11.9.4-ce.0 amd64
Environment details
- Operating System: Ubuntu 18.04.1 LTS
- Installation Target, remove incorrect values:
- Bare Metal Machine
- Installation Type, remove incorrect values:
- Upgrade from version 11.4
- Is this a single or multiple node installation?
- Single
- Resources
- CPU: Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
- Memory total: 377G
Configuration details
The pages related configuration is:
pages_external_url "http://pages.example.com/" gitlab_pages['enable'] = true gitlab_pages['access_control'] = true gitlab_pages['cert'] = nil; gitlab_pages['log_verbose'] = true gitlab_pages['inplace_chroot'] = true pages_nginx['enable'] = true pages_nginx['redirect_http_to_https'] = false pages_nginx['listen_port'] = 81; pages_nginx['listen_https'] = false;