Unable to renew/acquire certificates for custom domains

Summary

The GitLab instance is unable to renew/acquire certificates for custom domains with let's encrypt.

Steps to reproduce

1. create a pages deployment
2. add a new custom domain
3. complete verification
4. enable automatic let's encrypt certificates for HTTPS
5. Observe the background job for this task failing via the admin panel

For context, this occurs on a self-hosted CE instance. The domain where this occurred was cobalt.rocks, the verification passed, and it ran with this specific setup for the last few months. I'm unable to pin down when specifically it stopped renewing the certificate however it must be within the last 90 days, the underlying instance is updates asap for stable GitLab releases.

What is the current bug behavior?

The background job fails and is unable to acquire/renew a certificate. The UI for the project also doesn't show an error message but instead just shows the domain with no certificate.

What is the expected correct behavior?

Acquire/Renew a certificate and, if the error occurs, display an error to the user.

Relevant logs and/or screenshots

Job	PagesDomainSslRenewalWorker
Error Class	OpenSSL::X509::RequestError
Error Message	X509_REQ_set_version: passed invalid argument

Extra information:

 {"version"=>0, "store"=>nil, "trace_propagation_headers"=>{"sentry-trace"=>"9dd463f0d0ee4d5ebb455c59643323d0-02b495f037f94639", "baggage"=>"sentry-trace_id=9dd463f0d0ee4d5ebb455c59643323d0,sentry-release=eedc7c560c9"}, "meta.caller_id"=>"PagesDomainSslRenewalWorker", "correlation_id"=>"c8ce8ee1317a7cf1e66b669167c9f288", "meta.root_caller_id"=>"Cronjob", "meta.feature_category"=>"pages", "meta.project"=>"cobalt/homepage-v3", "meta.root_namespace"=>"cobalt", "meta.client_id"=>"ip/", "worker_data_consistency"=>"always", "worker_data_consistency_per_db"=>{"main"=>"always", "ci"=>"always"}, "size_limiter"=>"validated", "scheduled_at"=>1734827469.0100007, "idempotency_key"=>"resque:gitlab:duplicate:default:3e049b23492d45c80cd73b7a85eab84db50c6d1bcc74579926f0a735a7ebb67f"} 

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Debian 12
Current User:	git
Using RVM:	no
Ruby Version:	3.2.5
Gem Version:	3.5.23
Bundler Version:2.5.11
Rake Version:	13.0.6
Redis Version:	7.0.15
Sidekiq Version:7.2.4
Go Version:	unknown

GitLab information
Version:	17.7.0
Revision:	eedc7c560c9
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	14.11
URL:		https://git.cobalt.rocks
HTTP Clone URL:	https://git.cobalt.rocks/some-group/some-project.git
SSH Clone URL:	git@git.cobalt.rocks:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: openid_connect

GitLab Shell
Version:	14.39.0
Repository storages:
- default: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address: 	unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 	17.7.0
- default Git Version: 	2.47.0
- 

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.39.0 ? ... OK (14.39.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
1/2 ... yes
1/3 ... yes
1/4 ... yes
1/5 ... yes
1/6 ... yes
1/7 ... yes
1/8 ... yes
1/9 ... yes
1/10 ... yes
1/11 ... yes
1/12 ... yes
1/13 ... yes
1/14 ... yes
1/15 ... yes
1/16 ... yes
1/17 ... yes
1/18 ... yes
1/19 ... yes
1/20 ... yes
1/21 ... yes
1/22 ... yes
1/23 ... yes
1/24 ... yes
1/25 ... yes
1/26 ... yes
1/27 ... yes
1/29 ... yes
1/30 ... yes
1/31 ... yes
1/32 ... yes
1/33 ... yes
1/34 ... yes
1/35 ... yes
1/37 ... yes
1/38 ... yes
1/39 ... yes
1/40 ... yes
1/41 ... yes
5/43 ... yes
1/44 ... yes
5/45 ... yes
1/46 ... yes
1/47 ... yes
1/48 ... yes
1/51 ... yes
8/52 ... yes
1/53 ... yes
1/54 ... yes
1/55 ... yes
1/56 ... yes
12/58 ... yes
12/59 ... yes
1/62 ... yes
1/63 ... yes
1/64 ... yes
1/65 ... yes
1/71 ... yes
14/72 ... yes
14/74 ... yes
1/76 ... yes
1/77 ... yes
1/78 ... yes
12/79 ... yes
1/80 ... yes
14/82 ... yes
12/83 ... yes
5/84 ... yes
1/85 ... yes
1/86 ... yes
1/87 ... yes
1/88 ... yes
1/89 ... yes
1/90 ... yes
1/91 ... yes
1/92 ... yes
1/93 ... yes
5/94 ... yes
1/96 ... yes
1/97 ... yes
1/98 ... yes
1/99 ... yes
5/100 ... yes
5/101 ... yes
119/102 ... yes
1/104 ... yes
1/105 ... yes
119/106 ... yes
119/107 ... yes
119/108 ... yes
119/109 ... yes
1/110 ... yes
1/111 ... yes
119/112 ... yes
1/113 ... yes
132/114 ... yes
132/115 ... yes
132/116 ... yes
132/117 ... yes
132/118 ... yes
119/119 ... yes
5/120 ... yes
1/121 ... yes
132/122 ... yes
119/123 ... yes
119/124 ... yes
119/125 ... yes
1/126 ... yes
119/127 ... yes
1/128 ... yes
1/129 ... yes
1/130 ... yes
1/131 ... yes
119/132 ... yes
5/133 ... yes
5/134 ... yes
158/135 ... yes
132/136 ... yes
1/137 ... yes
132/138 ... yes
132/139 ... yes
5/141 ... yes
158/142 ... yes
132/143 ... yes
158/144 ... yes
158/145 ... yes
1/146 ... yes
1/147 ... yes
1/148 ... yes
5/149 ... yes
119/150 ... yes
132/151 ... yes
132/152 ... yes
179/153 ... yes
179/154 ... yes
179/155 ... yes
179/157 ... yes
179/158 ... yes
179/159 ... yes
179/160 ... yes
191/161 ... yes
191/162 ... yes
191/163 ... yes
191/164 ... yes
191/165 ... yes
191/166 ... yes
191/167 ... yes
191/168 ... yes
191/169 ... yes
191/170 ... yes
191/171 ... yes
191/172 ... yes
191/173 ... yes
191/174 ... yes
191/175 ... yes
193/176 ... yes
179/177 ... yes
179/178 ... yes
1/179 ... yes
1/180 ... yes
5/181 ... yes
1/182 ... yes
119/183 ... yes
1/185 ... yes
218/186 ... yes
1/187 ... yes
218/188 ... yes
218/189 ... yes
218/190 ... yes
1/191 ... yes
218/192 ... yes
1/193 ... yes
132/194 ... yes
1/195 ... yes
218/196 ... yes
218/197 ... yes
1/198 ... yes
1/199 ... yes
1/200 ... yes
236/201 ... yes
238/202 ... yes
1/203 ... yes
1/204 ... yes
1/205 ... yes
1/206 ... yes
1/207 ... yes
1/208 ... yes
1/209 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.2.5)
Git user has default SSH configuration? ... yes
Active users: ... 9
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

The error looks similar to https://github.com/ruby/openssl/issues/734, which was resolved in https://github.com/ruby/openssl/pull/747 and the fix was included in the latest release.