SSL certificates issues on some GitLab hosted pages with custom domains

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Since 20240312T1727Z I get PagerDuty alerts from HetrixTools for some GitLab pages with custom domains.

https://status.gitlab.com/ and https://twitter.com/gitlabstatus indicate there are no problems, but my local browsers, SSLabs and certcheckerapp indicate the autogenerated Let's Encrypt TLS certificate for one of the domains has expired for quite a while.

SSLabs is quite overloaded right now, so some of their links below can return a HTTP 503 response or time-out. That's why I included certcheckerapp links below as well: they they seem not overloaded.

These are the affected domains with corresponding GitLab pages projects and various checks.

martijn.pluimers.com domain

• https://martijn.pluimers.com - down for 30+ minutes (in the meantime 12+ hours) with an out-of-date (~1 month expired) TLS certificate 036e42b2684de2f4cb1f4e5a4e528622224703e0de247c4d37a3a13a5bd13f89
  • https://gitlab.com/wiert.me/private/web-sites/martijn.pluimers.com-ssl
    • https://gitlab.com/wiert.me/private/web-sites/martijn.pluimers.com-ssl/pages "Certificate: /CN=martijn.pluimers.com Expired", "Something went wrong while obtaining the Let's Encrypt certificate for martijn.pluimers.com. To retry visit your domain details."
    • https://gitlab.com/wiert.me/private/web-sites/martijn.pluimers.com-ssl/pages/domains/martijn.pluimers.com I retried the certificate from here, but on the …/pages link above it still lists "Expired" and on this page it re-shows "Something went wrong while obtaining the Let's Encrypt certificate." without a means to actually see what went wrong:
  • (non-SSL repository: https://gitlab.com/wiert.me/private/web-sites/martijn.pluimers.com)
  • Permanent failure reported from https://hetrixtools.com/report/uptime/6612c5ec1ec1c69bdae578a017f6c16c/
  • DNS entries are OK:
    • https://mxtoolbox.com/SuperTool.aspx?action=a%3amartijn.pluimers.com&run=toolpage CNAME martijn.pluimers.com wiert.me.gitlab.io correct as per …/domains/… page above: martijn.pluimers.com ALIAS wiert.me.gitlab.io.
    • https://mxtoolbox.com/SuperTool.aspx?action=txt%3a_gitlab-pages-verification-code.martijn.pluimers.com&run=toolpage TXT record is present and has correct information "gitlab-pages-verification-code=8a94bb0d8dcd8f908f004d14c2ac19d8" as per …/domains/… page above: _gitlab-pages-verification-code.martijn.pluimers.com TXT gitlab-pages-verification-code=8a94bb0d8dcd8f908f004d14c2ac19d8
  • Certificate is expired:
    • https://www.ssllabs.com/ssltest/analyze.html?d=martijn.pluimers.com indicating "grade T" (expired certificate)
    • https://certcheckerapp.com/home?hostname=martijn.pluimers.com&hideResult=false (certificate is expired)
      • https://certcheckerapp.com/certificate/036E42B2684DE2F4CB1F4E5A4E528622224703E0DE247C4D37A3A13A5BD13F89

ans.pluimers.com domain

• https://ans.pluimers.com - intermittent failure but still an up-to-date TLS certificate 708a50703e7dd6fc17b3d71f9e5662a12a0e5c12da432d46946b94201aea7834
  • https://gitlab.com/wiert.me/private/web-sites/ans.pluimers.com-ssl
    • https://gitlab.com/wiert.me/private/web-sites/ans.pluimers.com-ssl/pages "Certificate: /CN=ans.pluimers.com", "Something went wrong while obtaining the Let's Encrypt certificate for ans.pluimers.com. To retry visit your domain details."
      • https://gitlab.com/wiert.me/private/web-sites/ans.pluimers.com-ssl/pages/domains/ans.pluimers.com I have not retried renewing this one
  • (non-SSL repository: https://gitlab.com/wiert.me/private/web-sites/ans.pluimers.com)
  • Intermittent failures reported from https://hetrixtools.com/report/uptime/b7edda66fdd47ea7249be758210be081/
  • DNS entries are OK:
    • https://mxtoolbox.com/SuperTool.aspx?action=a%3aans.pluimers.com&run=toolpage CNAME ans.pluimers.com wiert.me.gitlab.io correct as per …/domains/… page above: ans.pluimers.com ALIAS wiert.me.gitlab.io.
    • https://mxtoolbox.com/SuperTool.aspx?action=txt%3a_gitlab-pages-verification-code.ans.pluimers.com&run=toolpage TXT record is present and has correct information "gitlab-pages-verification-code=bf54fc65162e9f747db70efaf34aa26f" as per …/pages/… page above: _gitlab-pages-verification-code.ans.pluimers.com TXT gitlab-pages-verification-code=bf54fc65162e9f747db70efaf34aa26f
  • Certificate is still valid:
    • https://www.ssllabs.com/ssltest/analyze.html?d=ans.pluimers.com indicates "grade A"
    • https://certcheckerapp.com/home?hostname=ans.pluimers.com&hideResult=false
      • https://certcheckerapp.com/certificate/708A50703E7DD6FC17B3D71F9E5662A12A0E5C12DA432D46946B94201AEA7834 certificate is OK

Remarks

• I know that most of the DNS servers for pluimers.com are down right now: a move that didn't work well. Working on that too, but given some health issues I can only do so much per day.
• The odd thing is that PagerDuty only started warning about the martijn.pluimers.com TLS certificate issue today while it has been expired 23 days ago on 2024-02-18.
• On hetrixtools, I have put ans.pluimers.com checking into "maintenance mode" to ensure I could sleep through the night (preventing 20+ PagerDuty phone calls)

My main questions

1. could the expired certificate on martijn.pluimers.com be a GitLab thing, or is this a DNS thing? (and if it is a DNS thing: what more than the above checks can I do?)
2. how can I get details on why a Let's Encrypt TLS certificate renewal fails?
3. are the intermittent time-out errors on ans.pluimers.com (see the hetrixtools link there) related to the certificate problem of martijn.pluimers.com?

Thanks in advance for getting back at this.

--jeroen