pages daemon fails DNS lookup (#4243) · Issues · GitLab.org / omnibus-gitlab

pages daemon fails DNS lookup

Summary

When enabling authentication for the pages daemon it tries to do a call back to the gitlab server and that fails when doing a DNS lookup to find the address of the gitlab server:

level=debug msg=“Fetching access token failed” error=“Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address” host=project.pages.example.com path="/auth?code=46c1f78&state=7uw%3D%3D"

The pages daemon works when removing the -daemon-* options:

-daemon-uid=998 -daemon-gid=998 -daemon-inplace-chroot=true

... and running this command line as git:

/opt/gitlab/embedded/bin/gitlab-pages -listen-proxy=localhost:8090 -pages-domain=pages.example.com -pages-root=/var/opt/gitlab/gitlab-rails/shared/pages -log-verbose -redirect-http=false -use-http2=true -artifacts-server=https://gitlab.example.com/api/v4 -artifacts-server-timeout=10 -auth-client-id=snip -auth-client-secret=snip -auth-redirect-uri=http://projects.pages.example.com/auth -auth-server=https://gitlab.example.com/ -auth-secret=snip -admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret -admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket

Steps to reproduce

Set up a gitlab server container using gitlab/gitlab-ce:11.9.4-ce.0
Put the gitlab container behind an external load balancer that terminates TLS and exposes https://gitlab.example.com
Enable pages with authentication according to the documentation and have the external LB expose its port as http://pages.example.com
Observe that it breaks.

What is the current bug behavior?

DNS lookup fails when making call back to the gitlab server for authentication

What is the expected correct behavior?

DNS lookup and authentication should work.

Relevant logs

2019-04-03_12:56:21.03194 time="2019-04-03T12:56:21Z" level=info msg="GitLab Pages Daemon" revision=869b94c version=1.5.0
2019-04-03_12:56:21.03196 time="2019-04-03T12:56:21Z" level=info msg="URL: https://gitlab.com/gitlab-org/gitlab-pages"
2019-04-03_12:56:21.03203 time="2019-04-03T12:56:21Z" level=debug msg="Start daemon with configuration" admin-https-cert= admin-https-key= admin-https-listener= admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket artifacts-server="https://gitlab.example.com/api/v4" artifacts-server-timeout=10 auth-client-id=$snip auth-client-secret=$snip auth-redirect-uri="http://projects.pages.example.com/auth" auth-secret=$snip auth-server="https://gitlab.example.com/" daemon-gid=998 daemon-inplace-chroot=true daemon-uid=998 default-config-filename=config disable-cross-origin-requests=false domain=pages.example.com listen-http= listen-https= listen-proxy="localhost:8090" log-format=text metrics-address= pages-domain=pages.example.com pages-root=/var/opt/gitlab/gitlab-rails/shared/pages pages-status= redirect-http=false root-cert= root-key= status_path= use-http-2=true
2019-04-03_12:56:21.03229 time="2019-04-03T12:56:21Z" level=debug msg="Set up proxy listener" listener="localhost:8090"
2019-04-03_12:56:21.03512 time="2019-04-03T12:56:21Z" level=debug msg="Set up admin unix socket" listener=/var/opt/gitlab/gitlab-pages/admin.socket
2019-04-03_12:56:21.03513 time="2019-04-03T12:56:21Z" level=info msg="running the daemon as unprivileged user" gid=998 in-place=true uid=998
2019-04-03_12:56:21.06698 time="2019-04-03T12:56:21Z" level=info msg="starting the daemon as unprivileged user" gid=998 uid=998
2019-04-03_12:56:21.07488 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=6.6999e-05 group=containers
2019-04-03_12:56:21.07491 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=2.5998e-05 group=gradle
2019-04-03_12:56:21.07492 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.209e-05 group=rm
2019-04-03_12:56:21.07494 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=1.0072e-05 [email protected]
2019-04-03_12:56:21.07495 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.4679e-05 group=statistics
2019-04-03_12:56:21.07505 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000112507 group=techsupport
2019-04-03_12:56:21.07530 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000341842 group=project
2019-04-03_12:56:21.07532 time="2019-04-03T12:56:21Z" level=debug msg="Configured domain" domain=project host=project.pages.example.com
2019-04-03_12:56:21.07533 time="2019-04-03T12:56:21Z" level=info msg="Updated all domains" count(domains)=1 duration=0.001172813 hash=d9836b4b6a5aeca375ed7c4718672245d57fb2a9fe1dd75cf9d9263ee503c0f93d517526366bd6a67180d6f2f1886ff6187f0fbca21119e87cca683f1e718d11
2019-04-03_13:10:42.72420 time="2019-04-03T13:10:42Z" level=debug msg="Authenticate request" host=project.pages.example.com path=/test/hest
2019-04-03_13:10:42.72421 time="2019-04-03T13:10:42Z" level=debug msg="No access token exists, redirecting user to OAuth2 login" host=project.pages.example.com path=/test/hest
2019-04-03_13:10:42.72434 project.pages.example.com 127.0.0.1:54244 - - [2019-04-03 13:10:42.723822327 +0000 UTC m=+861.664076193] "GET /test/hest HTTP/1.0" 302 127 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000491
2019-04-03_13:10:42.73268 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip"
2019-04-03_13:10:42.73271 time="2019-04-03T13:10:42Z" level=debug msg="User is authenticating via domain" domain="http://project.pages.example.com" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip"
2019-04-03_13:10:42.73276 projects.pages.example.com 127.0.0.1:54248 - - [2019-04-03 13:10:42.732566565 +0000 UTC m=+861.672820431] "GET /auth HTTP/1.0" 302 248 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000183
2019-04-03_13:10:42.82561 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.82562 time="2019-04-03T13:10:42Z" level=debug msg="Redirecting auth callback to custom domain" host=projects.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.82568 projects.pages.example.com 127.0.0.1:54264 - - [2019-04-03 13:10:42.825393741 +0000 UTC m=+861.765647616] "GET /auth HTTP/1.0" 302 163 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000260
2019-04-03_13:10:42.83338 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=project.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.83371 time="2019-04-03T13:10:42Z" level=error msg="failed to read SSL_CERT_FILE" error="open /opt/gitlab/embedded/ssl/certs/cacert.pem: no such file or directory"
2019-04-03_13:10:42.83435 time="2019-04-03T13:10:42Z" level=debug msg="Fetching access token failed" error="Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address" host=project.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.83442 project.pages.example.com 127.0.0.1:54268 - - [2019-04-03 13:10:42.832351673 +0000 UTC m=+861.772605539] "GET /auth HTTP/1.0" 503 2904 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.002045

Details of package version

Provide the package version installation details

gitlab-ce                                                     11.9.4-ce.0                         amd64

Environment details

Operating System: Ubuntu 18.04.1 LTS
Installation Target, remove incorrect values:
- Bare Metal Machine
Installation Type, remove incorrect values:
- Upgrade from version 11.4
Is this a single or multiple node installation?
Single
Resources
- CPU: Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
- Memory total: 377G

Configuration details

The pages related configuration is:

pages_external_url "http://pages.example.com/"
gitlab_pages['enable'] = true
gitlab_pages['access_control'] = true
gitlab_pages['cert'] = nil;
gitlab_pages['log_verbose'] = true
gitlab_pages['inplace_chroot'] = true
pages_nginx['enable'] = true
pages_nginx['redirect_http_to_https'] = false
pages_nginx['listen_port'] = 81;
pages_nginx['listen_https'] = false;

### Summary

When enabling authentication for the pages daemon it tries to do a call back to the gitlab server and that fails when doing a DNS lookup to find the address of the gitlab server:

```
level=debug msg=“Fetching access token failed” error=“Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address” host=project.pages.example.com path="/auth?code=46c1f78&state=7uw%3D%3D"
```

The pages daemon works when removing the -daemon-* options:
```
-daemon-uid=998 -daemon-gid=998 -daemon-inplace-chroot=true
```

... and running this command line as git:

```
/opt/gitlab/embedded/bin/gitlab-pages -listen-proxy=localhost:8090 -pages-domain=pages.example.com -pages-root=/var/opt/gitlab/gitlab-rails/shared/pages -log-verbose -redirect-http=false -use-http2=true -artifacts-server=https://gitlab.example.com/api/v4 -artifacts-server-timeout=10 -auth-client-id=snip -auth-client-secret=snip -auth-redirect-uri=http://projects.pages.example.com/auth -auth-server=https://gitlab.example.com/ -auth-secret=snip -admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret -admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket
```

### Steps to reproduce

* Set up a gitlab server container using gitlab/gitlab-ce:11.9.4-ce.0
* Put the gitlab container behind an external load balancer that terminates TLS and exposes https://gitlab.example.com
* Enable pages with authentication according to the documentation and have the external LB expose its port as http://pages.example.com
* Observe that it breaks.

### What is the current *bug* behavior?

* DNS lookup fails when making call back to the gitlab server for authentication

### What is the expected *correct* behavior?

* DNS lookup and authentication should work.

### Relevant logs

<details>
<summary> Relevant logs </summary>
<pre>
2019-04-03_12:56:21.03194 time="2019-04-03T12:56:21Z" level=info msg="GitLab Pages Daemon" revision=869b94c version=1.5.0
2019-04-03_12:56:21.03196 time="2019-04-03T12:56:21Z" level=info msg="URL: https://gitlab.com/gitlab-org/gitlab-pages"
2019-04-03_12:56:21.03203 time="2019-04-03T12:56:21Z" level=debug msg="Start daemon with configuration" admin-https-cert= admin-https-key= admin-https-listener= admin-secret-path=/var/opt/gitlab/gitlab-pages/admin.secret admin-unix-listener=/var/opt/gitlab/gitlab-pages/admin.socket artifacts-server="https://gitlab.example.com/api/v4" artifacts-server-timeout=10 auth-client-id=$snip auth-client-secret=$snip auth-redirect-uri="http://projects.pages.example.com/auth" auth-secret=$snip auth-server="https://gitlab.example.com/" daemon-gid=998 daemon-inplace-chroot=true daemon-uid=998 default-config-filename=config disable-cross-origin-requests=false domain=pages.example.com listen-http= listen-https= listen-proxy="localhost:8090" log-format=text metrics-address= pages-domain=pages.example.com pages-root=/var/opt/gitlab/gitlab-rails/shared/pages pages-status= redirect-http=false root-cert= root-key= status_path= use-http-2=true
2019-04-03_12:56:21.03229 time="2019-04-03T12:56:21Z" level=debug msg="Set up proxy listener" listener="localhost:8090"
2019-04-03_12:56:21.03512 time="2019-04-03T12:56:21Z" level=debug msg="Set up admin unix socket" listener=/var/opt/gitlab/gitlab-pages/admin.socket
2019-04-03_12:56:21.03513 time="2019-04-03T12:56:21Z" level=info msg="running the daemon as unprivileged user" gid=998 in-place=true uid=998
2019-04-03_12:56:21.06698 time="2019-04-03T12:56:21Z" level=info msg="starting the daemon as unprivileged user" gid=998 uid=998
2019-04-03_12:56:21.07488 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=6.6999e-05 group=containers
2019-04-03_12:56:21.07491 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=2.5998e-05 group=gradle
2019-04-03_12:56:21.07492 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.209e-05 group=rm
2019-04-03_12:56:21.07494 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=1.0072e-05 group=@pages.tmp
2019-04-03_12:56:21.07495 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=3.4679e-05 group=statistics
2019-04-03_12:56:21.07505 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000112507 group=techsupport
2019-04-03_12:56:21.07530 time="2019-04-03T12:56:21Z" level=debug msg="Loaded projects for group" duration=0.000341842 group=project
2019-04-03_12:56:21.07532 time="2019-04-03T12:56:21Z" level=debug msg="Configured domain" domain=project host=project.pages.example.com
2019-04-03_12:56:21.07533 time="2019-04-03T12:56:21Z" level=info msg="Updated all domains" count(domains)=1 duration=0.001172813 hash=d9836b4b6a5aeca375ed7c4718672245d57fb2a9fe1dd75cf9d9263ee503c0f93d517526366bd6a67180d6f2f1886ff6187f0fbca21119e87cca683f1e718d11
2019-04-03_13:10:42.72420 time="2019-04-03T13:10:42Z" level=debug msg="Authenticate request" host=project.pages.example.com path=/test/hest
2019-04-03_13:10:42.72421 time="2019-04-03T13:10:42Z" level=debug msg="No access token exists, redirecting user to OAuth2 login" host=project.pages.example.com path=/test/hest
2019-04-03_13:10:42.72434 project.pages.example.com 127.0.0.1:54244 - - [2019-04-03 13:10:42.723822327 +0000 UTC m=+861.664076193] "GET /test/hest HTTP/1.0" 302 127 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000491
2019-04-03_13:10:42.73268 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip"
2019-04-03_13:10:42.73271 time="2019-04-03T13:10:42Z" level=debug msg="User is authenticating via domain" domain="http://project.pages.example.com" host=projects.pages.example.com path="/auth?domain=http://project.pages.example.com&state=$snip"
2019-04-03_13:10:42.73276 projects.pages.example.com 127.0.0.1:54248 - - [2019-04-03 13:10:42.732566565 +0000 UTC m=+861.672820431] "GET /auth HTTP/1.0" 302 248 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000183
2019-04-03_13:10:42.82561 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=projects.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.82562 time="2019-04-03T13:10:42Z" level=debug msg="Redirecting auth callback to custom domain" host=projects.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.82568 projects.pages.example.com 127.0.0.1:54264 - - [2019-04-03 13:10:42.825393741 +0000 UTC m=+861.765647616] "GET /auth HTTP/1.0" 302 163 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.000260
2019-04-03_13:10:42.83338 time="2019-04-03T13:10:42Z" level=debug msg="Authentication callback" host=project.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.83371 time="2019-04-03T13:10:42Z" level=error msg="failed to read SSL_CERT_FILE" error="open /opt/gitlab/embedded/ssl/certs/cacert.pem: no such file or directory"
2019-04-03_13:10:42.83435 time="2019-04-03T13:10:42Z" level=debug msg="Fetching access token failed" error="Post https://gitlab.example.com/oauth/token: dial tcp: lookup gitlab.example.com on [::1]:53: dial udp [::1]:53: connect: cannot assign requested address" host=project.pages.example.com path="/auth?code=$snip%3D%3D"
2019-04-03_13:10:42.83442 project.pages.example.com 127.0.0.1:54268 - - [2019-04-03 13:10:42.832351673 +0000 UTC m=+861.772605539] "GET /auth HTTP/1.0" 503 2904 "https://gitlab.example.com/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/72.0.3626.121 Chrome/72.0.3626.121 Safari/537.36" 0.002045
</pre>
</details>

### Details of package version

<details>
<summary>Provide the package version installation details</summary>
<pre>
gitlab-ce                                                     11.9.4-ce.0                         amd64                               
</pre>
</details>

### Environment details

* Operating System: Ubuntu 18.04.1 LTS
* Installation Target, remove incorrect values:
  * Bare Metal Machine
* Installation Type, remove incorrect values:
  * Upgrade from version 11.4
* Is this a single or multiple node installation?
 * Single
* Resources
  * CPU: Intel(R) Xeon(R) CPU E5-2687W v3 @ 3.10GHz
  * Memory total: 377G

### Configuration details

<details>
The pages related configuration is:
<pre>

pages_external_url "http://pages.example.com/"
gitlab_pages['enable'] = true
gitlab_pages['access_control'] = true
gitlab_pages['cert'] = nil;
gitlab_pages['log_verbose'] = true
gitlab_pages['inplace_chroot'] = true
pages_nginx['enable'] = true
pages_nginx['redirect_http_to_https'] = false
pages_nginx['listen_port'] = 81;
pages_nginx['listen_https'] = false;

</pre>
</details>

Edited Apr 04, 2019 by Flemming Frandsen