Skip to content
GitLab
Next

pages daemon fails DNS lookup

Summary

When enabling authentication for the pages daemon it tries to do a call back to the gitlab server and that fails when doing a DNS lookup to find the address of the gitlab server:

level=debug msg=“Fetching access token failed” error=“Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address” host=project.pages.example.com path="/auth?code=46c1f78&state=7uw%3D%3D"

The pages daemon works when removing the -daemon-* options:

-daemon-uid=998 -daemon-gid=998 -daemon-inplace-chroot=true

... and running this command line as git:

/opt/gitlab/embedded/bin/gitlab-pages -listen-proxy=localhost:8090 -pages-domain=pages.example.com -pages-root=/var/opt/gitlab/gitlab-rails/shared/pages -log-verbose -redirect-http=false -use-http2=true -artifacts-server=https://gitlab.example.com/api/v4 -artifacts-server-timeout=10 -auth-client-id=snip -auth-client-secret=snip -auth-redirect-uri=http://projects.pages.example.com/auth -auth-server=https://gitlab.example.com/ -auth-secret=snip -admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret -admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket

Steps to reproduce

  • Set up a gitlab server container using gitlab/gitlab-ce:11.9.4-ce.0
  • Put the gitlab container behind an external load balancer that terminates TLS and exposes https://gitlab.example.com
  • Enable pages with authentication according to the documentation and have the external LB expose its port as http://pages.example.com
  • Observe that it breaks.

Workaround

See #4243 (comment 418829477) for details

cp /etc/resolv.conf /var/opt/gitlab/gitlab-rails/shared/pages/etc/resolv.conf
cp -rv /etc/ssl /var/opt/gitlab/gitlab-rails/shared/pages/etc/
gitlab-ctl restart gitlab-pages

What is the current bug behavior?

  • DNS lookup fails when making call back to the gitlab server for authentication

What is the expected correct behavior?

  • DNS lookup and authentication should work.

Relevant logs

Relevant logs 
2019-04-03_12:56:21.03194 time="2019-04-03T12:56:21Z" level=info msg="GitLab Pages Daemon" revision=869b94c version=1.5.0
2019-04-03_12:56:21.03196 time="2019-04-03T12:56:21Z" level=info msg="URL: https://gitlab.com/gitlab-org/gitlab-pages"
2019-04-03_12:56:21.03203 time="2019-04-03T12:56:21Z" level=debug msg="Start daemon with configuration" admin-https-cert= admin-https-key= admin-https-listener= admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket artifacts-server="https://gitlab.example.com/api/v4" artifacts-server-timeout=10 auth-client-id=$snip auth-client-secret=$snip auth-redirect-uri="http://projects.pages.example.com/auth" auth-secret=$snip auth-server="https://gitlab.example.com/" daemon-gid=998 daemon-inplace-chroot=true daemon-uid=998 default-config-filename=config disable-cross-origin-requests=false domain=pages.example.com listen-http= listen-https= listen-proxy="localhost:8090" log-format=text metrics-address= pages-domain=pages.example.com pages-root=/var/opt/gitlab/gitlab-rails/shared/pages pages-status= redirect-http=false root-cert= root-key= status_path= use-http-2=true
2019-04-03_12:56:21.03229 time="2019-04-03T12:56:21Z" level=debug msg="Set up proxy listener" listener="localhost:8090"
2019-04-03_12:56:21.03512 time="2019-04-03T12:56:21Z" level=debug msg="Set up admin unix socket" listener=/var/opt/gitlab/gitlab-pages/admin.socket
2019-04-03_12:56:21.03513 time="2019-04-03T12:56:21Z" level=info msg="running the daemon as unprivileged user" gid=998 in-place=true uid=998
2019-04-03_12:56:21.06698 time="2019-04-03T12:56:21Z" level=info msg="starting the daemon as unprivileged user" gid=998 uid=998
2019-04-03_12:56:21.07488 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=6.6999e-05 group=containers
2019-04-03_12:56:21.07491 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=2.5998e-05 group=gradle
2019-04-03_12:56:21.07492 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.209e-05 group=rm
2019-04-03_12:56:21.07494 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=1.0072e-05 group=@pages.tmp
2019-04-03_12:56:21.07495 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.4679e-05 group=statistics
2019-04-03_12:56:21.07505 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000112507 group=techsupport
2019-04-03_12:56:21.07530 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000341842 group=project
2019-04-03_12:56:21.07532 time="2019-04-03T12:56:21Z" level=debug msg="Configured domain" domain=project host=project.pages.example.com
2019-04-03_12:56:21.07533 time="2019-04-03T12:56:21Z" level=info msg="Updated all domains" count(domains)=1 duration=0.001172813 hash=d9836b4b6a5aeca375ed7c4718672245d57fb2a9fe1dd75cf9d9263ee503c0f93d517526366bd6a67180d6f2f1886ff6187f0fbca21119e87cca683f1e718d11
2019-04-03_13:10:42.72420 time="2019-04-03T13:10:42Z" level=debug msg="Authenticate request" host=project.pages.example.com path=/test/hest
2019-04-03_13:10:42.72421 time="2019-04-03T13:10:42Z" level=debug msg="No access token exists, redirecting user to OAuth2 login" host=project.pages.example.com path=/test/hest
2019-04-03_13:10:42.72434 project.pages.example.com 127.0.0.1:54244 - - [2019-04-03 13:10:42.723822327 +0000 UTC m=+861.664076193] "GET /test/hest HTTP/1.0" 302 127 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000491
2019-04-03_13:10:42.73268 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip"
2019-04-03_13:10:42.73271 time="2019-04-03T13:10:42Z" level=debug msg="User is authenticating via domain" domain="http://project.pages.example.com" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip"
2019-04-03_13:10:42.73276 projects.pages.example.com 127.0.0.1:54248 - - [2019-04-03 13:10:42.732566565 +0000 UTC m=+861.672820431] "GET /auth HTTP/1.0" 302 248 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000183
2019-04-03_13:10:42.82561 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.82562 time="2019-04-03T13:10:42Z" level=debug msg="Redirecting auth callback to custom domain" host=projects.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.82568 projects.pages.example.com 127.0.0.1:54264 - - [2019-04-03 13:10:42.825393741 +0000 UTC m=+861.765647616] "GET /auth HTTP/1.0" 302 163 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000260
2019-04-03_13:10:42.83338 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=project.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.83371 time="2019-04-03T13:10:42Z" level=error msg="failed to read SSL_CERT_FILE" error="open /opt/gitlab/embedded/ssl/certs/cacert.pem: no such file or directory"
2019-04-03_13:10:42.83435 time="2019-04-03T13:10:42Z" level=debug msg="Fetching access token failed" error="Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address" host=project.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.83442 project.pages.example.com 127.0.0.1:54268 - - [2019-04-03 13:10:42.832351673 +0000 UTC m=+861.772605539] "GET /auth HTTP/1.0" 503 2904 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.002045

Details of package version

Provide the package version installation details 
gitlab-ce                                                     11.9.4-ce.0                         amd64

Environment details

  • Operating System: Ubuntu 18.04.1 LTS
  • Installation Target, remove incorrect values:
    • Bare Metal Machine
  • Installation Type, remove incorrect values:
    • Upgrade from version 11.4
  • Is this a single or multiple node installation?
  • Single
  • Resources
    • CPU: Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
    • Memory total: 377G

Configuration details

The pages related configuration is: 
pages_external_url "http://pages.example.com/"
gitlab_pages['enable'] = true
gitlab_pages['access_control'] = true
gitlab_pages['cert'] = nil;
gitlab_pages['log_verbose'] = true
gitlab_pages['inplace_chroot'] = true
pages_nginx['enable'] = true
pages_nginx['redirect_http_to_https'] = false
pages_nginx['listen_port'] = 81;
pages_nginx['listen_https'] = false;
Edited by Jaime Martinez