Let's Encrypt failing to generate certificates
Problem
I am trying to install the latest version of GitLab, and am getting an error when trying to auto-generate Let's Encrypt certificates.
There seems to be an issue with the server challenging for authentication (I assume rails) rather than responding to the ACME challenge.
Logs:
Recipe: letsencrypt::enable
* ruby_block[http external-url] action run (skipped due to only_if)
* directory[/etc/gitlab/ssl] action create (up to date)
* directory[/var/log/gitlab/lets-encrypt] action create (up to date)
* acme_selfsigned[gitlab.gitlab-ci.com] action create
* file[gitlab.gitlab-ci.com SSL selfsigned key] action create_if_missing (up to date)
* file[gitlab.gitlab-ci.com SSL selfsigned crt] action create_if_missing (up to date)
* file[gitlab.gitlab-ci.com SSL selfsigned chain] action create_if_missing (skipped due to not_if)
(up to date)
Recipe: letsencrypt::http_authorization
* letsencrypt_certificate[gitlab.gitlab-ci.com] action create
* acme_certificate[staging] action create
* file[gitlab.gitlab-ci.com SSL key] action nothing (skipped due to action :nothing)
* file[gitlab.gitlab-ci.com SSL key] action create_if_missing (up to date)
* directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action nothing (skipped due to action :nothing)
* directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
* file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] action nothing (skipped due to action :nothing)
* file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] action create[2023-03-13T20:52:56+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] created file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY
- create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY[2023-03-13T20:52:56+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] updated file contents /var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY
- update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY from none to 60552e
--- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY 2023-03-13 20:52:56.091472730 +0000
+++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY20230313-23022-1me6naj 2023-03-13 20:52:56.091472730 +0000
@@ -1 +1,2 @@
+YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY.b2HHgxIVMiRcWA-dU75WCmzU8PGdjNOMk7o8nxUei0o[2023-03-13T20:52:56+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] owner changed to 0
[2023-03-13T20:52:56+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] group changed to 0
[2023-03-13T20:52:56+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] mode changed to 644
- change mode from '' to '0644'
- change owner from '' to 'root'
- change group from '' to 'root'
* file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] action delete[2023-03-13T20:52:57+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY] deleted file at /var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY
- delete file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/YiBfWGydWd0zzxrzUYgbkrTSpEW_wW5QL0O7k_uDzfY
* ruby_block[create certificate for gitlab.gitlab-ci.com] action run
================================================================================
Error executing action `run` on resource 'ruby_block[create certificate for gitlab.gitlab-ci.com]'
================================================================================
RuntimeError
------------
[gitlab.gitlab-ci.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5710648024/2cSAkg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"35.202.30.142: Invalid response from http://gitlab.gitlab-ci.com/users/sign_in: \"<!DOCTYPE html>\\n<html class=\\\"devise-layout-html\\\">\\n<head prefix=\\\"og: http://ogp.me/ns#\\\">\\n<meta charset=\\\"utf-8\\\">\\n<title>Sign in ·\"", "status"=>403}} ]
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb
110: ruby_block "create certificate for #{new_resource.cn}" do
111: block do
112: unless (all_validations.map { |authz| authz.status == 'valid' }).all?
113: errors = all_validations.select { |authz| authz.status != 'valid' }.map do |authz|
114: "{url: #{authz.url}, status: #{authz.status}, error: #{authz.error}} "
115: end.reduce(:+)
116:
117: raise "[#{new_resource.cn}] Validation failed, unable to request certificate, Errors: [#{errors}]"
118: end
119:
120: begin
121: newcert = acme_cert(order, new_resource.cn, mykey, new_resource.alt_names)
122: rescue Acme::Client::Error => e
123: raise "[#{new_resource.cn}] Certificate request failed: #{e.message}"
124: else
125: Chef::Resource::File.new("#{new_resource.cn} SSL new crt", run_context).tap do |f|
126: f.path new_resource.crt
127: f.owner new_resource.owner
128: f.group new_resource.group
129: f.content newcert
130: f.mode 00644
131: end.run_action :create
132: end
133: end
134: end
135: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
ruby_block("create certificate for gitlab.gitlab-ci.com") do
action [:run]
default_guard_interpreter :default
declared_type :ruby_block
cookbook_name "letsencrypt"
recipe_name "http_authorization"
block #<Proc:0x00007f354d1c9600 /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111>
end
System Info:
------------
chef_version=17.10.0
platform=debian
platform_version=11
ruby=ruby 2.7.7p221 (2022-11-24 revision 168ec2b1e5) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
================================================================================
Error executing action `create` on resource 'acme_certificate[staging]'
================================================================================
RuntimeError
------------
ruby_block[create certificate for gitlab.gitlab-ci.com] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.gitlab-ci.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5710648024/2cSAkg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"35.202.30.142: Invalid response from http://gitlab.gitlab-ci.com/users/sign_in: \"<!DOCTYPE html>\\n<html class=\\\"devise-layout-html\\\">\\n<head prefix=\\\"og: http://ogp.me/ns#\\\">\\n<meta charset=\\\"utf-8\\\">\\n<title>Sign in ·\"", "status"=>403}} ]
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'
Resource Declaration:
---------------------
suppressed sensitive resource output
Compiled Resource:
------------------
suppressed sensitive resource output
System Info:
------------
chef_version=17.10.0
platform=debian
platform_version=11
ruby=ruby 2.7.7p221 (2022-11-24 revision 168ec2b1e5) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
================================================================================
Error executing action `create` on resource 'letsencrypt_certificate[gitlab.gitlab-ci.com]'
================================================================================
RuntimeError
------------
acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for gitlab.gitlab-ci.com] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.gitlab-ci.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5710648024/2cSAkg, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"35.202.30.142: Invalid response from http://gitlab.gitlab-ci.com/users/sign_in: \"<!DOCTYPE html>\\n<html class=\\\"devise-layout-html\\\">\\n<head prefix=\\\"og: http://ogp.me/ns#\\\">\\n<meta charset=\\\"utf-8\\\">\\n<title>Sign in ·\"", "status"=>403}} ]
Cookbook Trace: (most recent call first)
----------------------------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:117:in `block (3 levels) in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:110:in `block in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb:43:in `block in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb
6: letsencrypt_certificate site do
7: crt node['gitlab']['nginx']['ssl_certificate']
8: key node['gitlab']['nginx']['ssl_certificate_key']
9: notifies :run, "execute[reload nginx]", :immediate
10: notifies :run, 'ruby_block[display_le_message]'
11: only_if { omnibus_helper.service_up?('nginx') }
12: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:6:in `from_file'
letsencrypt_certificate("gitlab.gitlab-ci.com") do
action [:create]
updated true
updated_by_last_action true
default_guard_interpreter :default
declared_type :letsencrypt_certificate
cookbook_name "letsencrypt"
recipe_name "http_authorization"
crt "/etc/gitlab/ssl/gitlab.gitlab-ci.com.crt"
key "/etc/gitlab/ssl/gitlab.gitlab-ci.com.key"
alt_names []
cn "gitlab.gitlab-ci.com"
only_if { #code block }
end
System Info:
------------
chef_version=17.10.0
platform=debian
platform_version=11
ruby=ruby 2.7.7p221 (2022-11-24 revision 168ec2b1e5) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/cinc-client
executable=/opt/gitlab/embedded/bin/cinc-client
[2023-03-13T20:52:57+00:00] INFO: Running queued delayed notifications before re-raising exception
[2023-03-13T20:52:57+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending restart action to runit_service[puma] (delayed)
Recipe: gitlab::puma
runit_service[puma] action restart (up to date)
[2023-03-13T20:52:57+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending restart action to sidekiq_service[sidekiq] (delayed)
Recipe: gitlab::sidekiq
sidekiq_service[sidekiq] action restart
service[sidekiq] action nothing (skipped due to action :nothing)
runit_service[sidekiq] action restart (up to date)
(up to date)
[2023-03-13T20:53:00+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
Recipe: gitlab::gitlab-rails
execute[clear the gitlab-rails cache] action run[2023-03-13T20:53:56+00:00] INFO: execute[clear the gitlab-rails cache] ran successfully