Skip to content

gitlab-pages ingores new tls certificate (except shortly after restart)

Summary

gitlab-pages (in a docker-compose setup) in gitlab-ee only uses the new tls certificate for ~2 minutes before reverting back to the previous one.

Steps to reproduce

  1. Run gitlab-ee from a docker
  2. Use gitlab_pages to serve tls directly (with pages_nginx['enable'] = false)
  3. Use cert-files from outside the docker mounted with volumes, which link to the actual files
  4. Update the certificate
  5. try to reload the certificate by restarting the container with docker compose down and docker compose up -d

What is the current bug behavior?

After starting the docker with docker compose up -d, gitlab-pages uses the new certificate for pages for ~2 min (very roughly) before serving the previous one again.

What is the expected correct behavior?

gitlab-pages only serves the new certificate and it can be loaded with some of the options above.

Further Details

I am not able to load the new certificate directly, using either of

  • gitlab-ctl hup gitlab-pages
  • gitlab-ctl restart gitlab-pages
  • gitlab-ctl reconfigure
  • gitlab-ctl restart nginx (commands are run inside the docker)

Other things

  • Testing the certificate with /opt/gitlab/embedded/bin/openssl x509 -in /etc/letsencrypt/live/pages.[...]/fullchain.pem -text -noout works and shows the new certificate.
  • Making either of the key files invalid is detected by gitlab-pages when restarted, leading to gitlab-pages failing to start.
  • I first saw this problem on version gitlab-ee 18.1.1, it was not resolved by updating though.
  • I tried deleting the trusted-certs-hash (rm /var/opt/gitlab/trusted-certs-directory-hash)
  • or adding the cert to /etc/gitlab/trusted-certs

Relevant logs and/or screenshots

$ docker compose up -d
[+] Running 2/2
 ✔ Network hellogitty_default   Created                                                              0.0s
 ✔ Container hellogitty_gitlab  Started                                                              0.2s
# a bit of time past
$ date
Wed Sep  3 02:30:25 PM CEST 2025
$ openssl s_client -connect pages.[...]:443 -servername pages.[...] </dev/null | grep notAfter
depth=2 C = GR, O = Hellenic Academic and Research Institutions CA, CN = HARICA TLS ECC Root CA 2021
verify return:1
depth=1 C = GR, O = Hellenic Academic and Research Institutions CA, CN = GEANT TLS ECC 1
verify return:1
depth=0 C = AT, ST = Wien, O = Technische Universit\C3\A4t Wien, CN = pages.[...]
verify return:1
DONE
$ date
Wed Sep  3 02:31:32 PM CEST 2025
$ openssl s_client -connect pages.[...]:443 -servername pages.[...] </dev/null | grep notAfter
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority
verify return:1
depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo ECC Organization Validation Secure Server CA
verify return:1
depth=0 C = AT, ST = Wien, O = Technische Universit\C3\A4t Wien, CN = pages.[...]
verify error:num=10:certificate has expired
notAfter=Aug 29 23:59:59 2025 GMT
verify return:1
depth=0 C = AT, ST = Wien, O = Technische Universit\C3\A4t Wien, CN = pages.[...]
notAfter=Aug 29 23:59:59 2025 GMT
verify return:1
DONE
Expand for output from `gitlab-ctl tail gitlab-pages` during gitlab startup 
tail: '/var/log/gitlab/gitlab-pages/current' has become inaccessible: No such file or directory
tail: '/var/log/gitlab/gitlab-pages/current' has appeared;  following new file
{"level":"info","msg":"GitLab Pages","revision":"d62f04e6","time":"2025-09-03T13:44:58Z","version":"18.3.1"}
{"level":"info","msg":"URL: https://gitlab.com/gitlab-org/gitlab-pages","time":"2025-09-03T13:44:58Z"}
{"config_addr":"0.0.0.0:81","level":"info","listen_addr":{"IP":"::","Port":81,"Zone":""},"msg":"server listening on: [::]:81","time":"2025-09-03T13:44:58Z"}
{"config_addr":"localhost:8090","level":"info","listen_addr":{"IP":"127.0.0.1","Port":8090,"Zone":""},"msg":"server listening on: 127.0.0.1:8090","time":"2025-09-03T13:44:58Z"}
{"config_addr":"0.0.0.0:444","level":"info","listen_addr":{"IP":"::","Port":444,"Zone":""},"msg":"server listening on: [::]:444","time":"2025-09-03T13:44:58Z"}
{"level":"info","msg":"http: TLS handshake error from 128.130.171.200:43578: write tcp 172.19.0.2:444-\u003e128.130.171.200:43578: i/o timeout","time":"2025-09-03T13:45:03Z"}
{"level":"info","msg":"http: TLS handshake error from 128.131.171.110:57506: write tcp 172.19.0.2:444-\u003e128.131.171.110:57506: i/o timeout","time":"2025-09-03T13:45:03Z"}
{"correlation_id":"","error":"can't extract namespace from path because first segment is empty","host":"pages.par.tuwien.ac.at","level":"error","msg":"can't convert URL","orig_host":"pages.par.tuwien.ac.at","orig_path":"/","pages_domain":"pages.par.tuwien.ac.at","path":"/","time":"2025-09-03T13:45:06Z"}
{"correlation_id":"01K47X5VVYQRRNYNW01VP8V0CN","error":"HTTP status: 502","host":"pages.par.tuwien.ac.at","level":"error","msg":"could not fetch domain information from a source","path":"/","time":"2025-09-03T13:45:06Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01K47X5VVYQRRNYNW01VP8V0CN","duration_ms":0,"host":"pages.par.tuwien.ac.at","level":"info","method":"GET","msg":"access","pages_https":true,"proto":"HTTP/2.0","read_bytes":116,"referrer":"","remote_addr":"128.130.171.200:36912","remote_ip":"128.130.171.200","status":502,"system":"http","time":"2025-09-03T13:45:06Z","ttfb_ms":0,"uri":"/","user_agent":"Blackbox Exporter/0.23.0","written_bytes":3563}
{"level":"info","msg":"http: TLS handshake error from 128.130.171.200:55612: remote error: tls: bad certificate","time":"2025-09-03T13:45:15Z"}
{"correlation_id":"","error":"can't extract namespace from path because first segment is empty","host":"pages.par.tuwien.ac.at","level":"error","msg":"can't convert URL","orig_host":"pages.par.tuwien.ac.at","orig_path":"/","pages_domain":"pages.par.tuwien.ac.at","path":"/","time":"2025-09-03T13:45:21Z"}
{"correlation_id":"01K47X6AGPZVE864MVHJ0FMWV4","error":"HTTP status: 502","host":"pages.par.tuwien.ac.at","level":"error","msg":"could not fetch domain information from a source","path":"/","time":"2025-09-03T13:45:21Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01K47X6AGPZVE864MVHJ0FMWV4","duration_ms":0,"host":"pages.par.tuwien.ac.at","level":"info","method":"GET","msg":"access","pages_https":true,"proto":"HTTP/2.0","read_bytes":116,"referrer":"","remote_addr":"128.130.171.200:55628","remote_ip":"128.130.171.200","status":502,"system":"http","time":"2025-09-03T13:45:21Z","ttfb_ms":0,"uri":"/","user_agent":"Blackbox Exporter/0.23.0","written_bytes":3563}
{"level":"info","msg":"http: TLS handshake error from 128.130.171.200:51086: remote error: tls: bad certificate","time":"2025-09-03T13:45:30Z"}
{"correlation_id":"","error":"can't extract namespace from path because first segment is empty","host":"pages.par.tuwien.ac.at","level":"error","msg":"can't convert URL","orig_host":"pages.par.tuwien.ac.at","orig_path":"/","pages_domain":"pages.par.tuwien.ac.at","path":"/","time":"2025-09-03T13:45:36Z"}
{"correlation_id":"01K47X6S5FP52C6Z389EA21PKB","error":"HTTP status: 502","host":"pages.par.tuwien.ac.at","level":"error","msg":"could not fetch domain information from a source","path":"/","time":"2025-09-03T13:45:36Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01K47X6S5FP52C6Z389EA21PKB","duration_ms":0,"host":"pages.par.tuwien.ac.at","level":"info","method":"GET","msg":"access","pages_https":true,"proto":"HTTP/2.0","read_bytes":116,"referrer":"","remote_addr":"128.130.171.200:50872","remote_ip":"128.130.171.200","status":502,"system":"http","time":"2025-09-03T13:45:36Z","ttfb_ms":0,"uri":"/","user_agent":"Blackbox Exporter/0.23.0","written_bytes":3563}
{"level":"info","msg":"http: TLS handshake error from 128.130.171.200:46114: remote error: tls: bad certificate","time":"2025-09-03T13:45:45Z"}
{"correlation_id":"","error":"can't extract namespace from path because first segment is empty","host":"pages.par.tuwien.ac.at","level":"error","msg":"can't convert URL","orig_host":"pages.par.tuwien.ac.at","orig_path":"/","pages_domain":"pages.par.tuwien.ac.at","path":"/","time":"2025-09-03T13:45:51Z"}
{"correlation_id":"01K47X77T79X4MAGA3NA7FA88H","error":"HTTP status: 502","host":"pages.par.tuwien.ac.at","level":"error","msg":"could not fetch domain information from a source","path":"/","time":"2025-09-03T13:45:51Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01K47X77T79X4MAGA3NA7FA88H","duration_ms":0,"host":"pages.par.tuwien.ac.at","level":"info","method":"GET","msg":"access","pages_https":true,"proto":"HTTP/2.0","read_bytes":116,"referrer":"","remote_addr":"128.130.171.200:46126","remote_ip":"128.130.171.200","status":502,"system":"http","time":"2025-09-03T13:45:51Z","ttfb_ms":0,"uri":"/","user_agent":"Blackbox Exporter/0.23.0","written_bytes":3563}
{"level":"info","msg":"GitLab Pages","revision":"d62f04e6","time":"2025-09-03T13:45:53Z","version":"18.3.1"}
{"level":"info","msg":"URL: https://gitlab.com/gitlab-org/gitlab-pages","time":"2025-09-03T13:45:53Z"}
{"config_addr":"0.0.0.0:81","level":"info","listen_addr":{"IP":"::","Port":81,"Zone":""},"msg":"server listening on: [::]:81","time":"2025-09-03T13:45:53Z"}
{"config_addr":"0.0.0.0:444","level":"info","listen_addr":{"IP":"::","Port":444,"Zone":""},"msg":"server listening on: [::]:444","time":"2025-09-03T13:45:53Z"}
{"config_addr":"localhost:8090","level":"info","listen_addr":{"IP":"127.0.0.1","Port":8090,"Zone":""},"msg":"server listening on: 127.0.0.1:8090","time":"2025-09-03T13:45:53Z"}
{"correlation_id":"","error":"URL does not match pages domain","host":"wiki.par.tuwien.ac.at","level":"error","msg":"can't convert URL","orig_host":"wiki.par.tuwien.ac.at","orig_path":"/","pages_domain":"pages.par.tuwien.ac.at","path":"/","time":"2025-09-03T13:46:00Z"}
{"Namespace in path":"","Request host":"wiki.par.tuwien.ac.at","Session host":null,"correlation_id":"01K47X7H2VZS9Q69R7W479ZPMW","host":"wiki.par.tuwien.ac.at","level":"info","msg":"Resetting session values","path":"/","state":"","time":"2025-09-03T13:46:00Z"}
{"Namespace in path":"","Request host":"wiki.par.tuwien.ac.at","Session host":null,"correlation_id":"01K47X7H2VZS9Q69R7W479ZPMW","host":"wiki.par.tuwien.ac.at","level":"info","msg":"Resetting session values","path":"/","state":"","time":"2025-09-03T13:46:00Z"}
{"Namespace in path":"","Request host":"wiki.par.tuwien.ac.at","Session host":null,"correlation_id":"01K47X7H2VZS9Q69R7W479ZPMW","host":"wiki.par.tuwien.ac.at","level":"info","msg":"Resetting session values","path":"/","state":"","time":"2025-09-03T13:46:00Z"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01K47X7H2VZS9Q69R7W479ZPMW","duration_ms":0,"host":"wiki.par.tuwien.ac.at","level":"info","method":"GET","msg":"access","pages_https":true,"proto":"HTTP/2.0","read_bytes":114,"referrer":"","remote_addr":"128.130.171.200:46646","remote_ip":"128.130.171.200","status":302,"system":"http","time":"2025-09-03T13:46:00Z","ttfb_ms":0,"uri":"/","user_agent":"Blackbox Exporter/0.23.0","written_bytes":164}
{"level":"info","msg":"http: TLS handshake error from 128.130.171.200:46656: remote error: tls: bad certificate","time":"2025-09-03T13:46:01Z"}

Results of GitLab environment info

Expand for output related to GitLab environment info 
# docker-compose.yml
services:
  web:
    image: 'gitlab/gitlab-ee:18.3.1-ee.0'
    container_name: hellogitty_gitlab
    restart: unless-stopped
    hostname: 'hellogitty.[...]'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://hellogitty.[...]'
        nginx['ssl_certificate'] = "/etc/letsencrypt/live/hellogitty.[...]/fullchain.pem"
        nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/hellogitty.[...]/privkey.pem"
        nginx['listen_addresses'] = ['0.0.0.0'] # The primary IP of the GitLab instance

        pages_external_url 'https://pages.[...]'
        pages_nginx['enable'] = false
        gitlab_pages['external_http'] = ['0.0.0.0:81'] # The secondary IPs for the GitLab Pages daemon
        gitlab_pages['external_https'] = ['0.0.0.0:444']
        gitlab_pages['redirect_http'] = true
        gitlab_pages['cert'] = "/etc/letsencrypt/live/pages.[...]/fullchain.pem"
        gitlab_pages['cert_key'] = "/etc/letsencrypt/live/pages.[...]/privkey.pem"

        gitlab_pages['access_control'] = true
        gitlab_pages["namespace_in_path"] = true

        prometheus['enable'] = false
    ports:
      - '[...].111:80:80'
      - '[...].111:443:443'
      - '[...].111:22:22'
      - '[...].112:80:81'
      - '[...].112:443:444'
    volumes:
      - '/etc/hellogitty:/etc/gitlab'
      - '/var/logs/hellogitty:/var/log/gitlab'
      - '/srv/hellogitty:/var/opt/gitlab'
      - './backups/:/var/opt/gitlab/backups/'
      - '/etc/letsencrypt/live/hellogitty.[...]/:/etc/letsencrypt/live/hellogitty.[...]/:ro'
      - '/etc/letsencrypt/archive/hellogitty.[...]/:/etc/letsencrypt/archive/hellogitty.[...]/:ro'
      - '/etc/letsencrypt/live/pages.[...]/:/etc/letsencrypt/live/pages.[...]/:ro'
      - '/etc/letsencrypt/archive/pages.[...]/:/etc/letsencrypt/archive/pages.[...]/:ro'
    shm_size: '256m'
    logging:
      options:
        max-size: '50m'

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)


System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	3.2.8
Gem Version:	3.6.9
Bundler Version:2.7.1
Rake Version:	13.0.6
Redis Version:	7.2.10
Sidekiq Version:7.3.9
Go Version:	unknown

GitLab information
Version:	18.3.1-ee
Revision:	cb15859792b
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	16.8
URL:		https://hellogitty.[...]
HTTP Clone URL:	https://hellogitty.[...]/some-group/some-project.git
SSH Clone URL:	git@hellogitty.[...]:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	14.44.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	18.3.1
- default Git Version: 	2.50.1.gl1

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:check SANITIZE=true`)


Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 14.44.0 ? ... OK (14.44.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
2/2 ... yes
3/4 ... yes
10/6 ... yes
10/7 ... yes
10/8 ... yes
14/11 ... yes
13/12 ... yes
3/14 ... yes
3/15 ... yes
3/16 ... yes
3/17 ... yes
10/18 ... yes
10/19 ... yes
3/20 ... yes
3/22 ... yes
10/23 ... yes
14/24 ... yes
3/25 ... yes
16/26 ... yes
16/27 ... yes
10/29 ... yes
10/30 ... yes
9/32 ... yes
10/33 ... yes
19/34 ... yes
14/35 ... yes
9/36 ... yes
10/37 ... yes
14/38 ... yes
14/39 ... yes
14/40 ... yes
19/41 ... yes
3/42 ... yes
14/43 ... yes
14/44 ... yes
19/45 ... yes
14/46 ... yes
14/47 ... yes
14/48 ... yes
14/49 ... yes
19/50 ... yes
19/51 ... yes
3/52 ... yes
10/53 ... yes
14/54 ... yes
10/56 ... yes
10/57 ... yes
10/58 ... yes
70/59 ... yes
70/60 ... yes
10/61 ... yes
10/62 ... yes
3/63 ... yes
3/64 ... yes
10/65 ... yes
10/67 ... yes
128/69 ... yes
128/71 ... yes
128/73 ... yes
128/74 ... yes
142/75 ... yes
142/76 ... yes
10/77 ... yes
128/78 ... yes
142/79 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.2.8)
Git user has default SSH configuration? ... yes
Active users: ... 18
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-9.x or OpenSearch version 1.x-3.x ... skipped (advanced search is disabled)
All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Patch release information for backports

If the bug fix needs to be backported in a patch release to a version under the maintenance policy, please follow the steps on the patch release runbook for GitLab engineers.

Refer to the internal "Release Information" dashboard for information about the next patch release, including the targeted versions, expected release date, and current status.

High-severity bug remediation

To remediate high-severity issues requiring an internal release for single-tenant SaaS instances, refer to the internal release process for engineers.

Edited by 🤖 GitLab Bot 🤖