Gitlab Pages throw 502 error in Docker environment after update from 13.12 to 14.02
Summary
Yesterday night, we updated our Dockerized Gitlab CE environment to 14.02.
After the update to 14.01, the pages are gone and throw an error 502. We then updated to 14.02 to check, if there is an fix included, but then stop updating and check the reason.
There is also an entry within the forun: https://forum.gitlab.com/t/pages-error-502/58477
What is the current bug behavior?
Pages show 502 error
What is the expected correct behavior?
Pages show page content
Relevant logs and/or screenshots
/var/log/gitlab/gitlab-pages/current
{"correlation_id":"01FG7VMKWBA7TR8P72XD3JSZCG","error":null,"level":"debug","lookup_error":"Get \"https://git.redoo-networks.com/api/v4/internal/pages?host=custompage.domain.com\": dial tcp 192.168.48.16:443: c
onnect: connection refused","lookup_name":"custompage.domain.com","lookup_paths":null,"msg":"retrieval response sent","requested_domain":"custompage.domain.com","time":"2021-09-22T22:57:23Z"}
{"correlation_id":"01FG7VMKWBA7TR8P72XD3JSZCG","error":"Get \"https://git.redoo-networks.com/api/v4/internal/pages?host=custompage.domain.com\": dial tcp 192.168.48.16:443: connect: connection refused","host":"
custompage.domain.com","level":"error","msg":"could not fetch domain information from a source","path":"/app/sessions/signIn","time":"2021-09-22T22:57:23Z"}
Results of GitLab environment info
We use dockerized Gitlab behind a Traefik reverse proxy, including SSL Termination. The gitlab docker environment that's why is configured with "nginx['listen_https'] = false", like you recommend here: https://docs.gitlab.com/omnibus/settings/nginx.html#external-proxy-and-load-balancer-ssl-termination
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.6 Redis Version: 6.0.14 Git Version: 2.32.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 14.2.4 Revision: 25bed1192b4 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.8 URL: https://git.redoo-networks.com HTTP Clone URL: https://git.redoo-networks.com/some-group/some-project.git SSH Clone URL: ssh://git@git.redoo-networks.com:2233/some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.19.1 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 20 users of 100 limit.
Checking LDAP ... Finished
Possible fixes
Because https is disabled in nginx, he only listen on port 80, like you propose in docs.
But within the gitlab docker container, you, or the based ubuntu image, pin the local ip to hostname within /etc/hosts:
192.168.48.16 git.redoo-networks.com git
When the gitlab-pages daemon now tries to contact "https://git.redoo-networks.com/...", he get a "Connected refused", because local IP is only listing on http port 80. We found a quick workaround to modify /etc/hosts and set our gitlab hostname to the IP of traefik proxy. (192.168.48.13) So the request was possible and pages come back. But this is only a temporarily solution, because after restart the ip is set back to normal/internal one.
We will enable nginx['listen_https'] during next maintenance window next night to check, if this solves the problem permanently.
But then I would recommend, you adjust your docs.
Another possible fix also would be to configure gitlab-pages to do requests to gitlab over http (but insecure)