Error configuring let's encrypt
Summary
I can't reconfigure after activating let's encrypt
Steps to reproduce
I did the initial let's encrypt conf described here https://docs.gitlab.com/omnibus/settings/ssl.html#primary-gitlab-instance.
Then I ran the reconfigure command.
IUt should be noted that i previously tried to manually configure the certs a long ago. But i followed this blogpost to remove any old config: https://community.letsencrypt.org/t/how-to-remove-old-bad-configuration/13237/8
What is the current bug behavior?
Reconfigure end up with the error There was an error running gitlab-ctl reconf igure:
letsencrypt_certificate[git.doamin.com] (letsencrypt::http_authorization line 5) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for git.domain.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.domain.com] Validation failed, unable to request certificate
What is the expected correct behavior?
Reconfigure and support ssl
Relevant logs and/or screenshots
output of the gitlab-ctl renew-le-certs command:
Starting Chef Client, version 14.13.11
resolving cookbooks for run list: ["gitlab::letsencrypt_renew"]
Synchronizing Cookbooks:
- gitlab (0.0.1)
- package (0.1.0)
- postgresql (0.1.0)
- redis (0.1.0)
- monitoring (0.1.0)
- registry (0.1.0)
- mattermost (0.1.0)
- consul (0.1.0)
- gitaly (0.1.0)
- praefect (0.1.0)
- letsencrypt (0.1.0)
- nginx (0.1.0)
- runit (4.3.0)
- acme (4.0.0)
- crond (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 14 resources
Recipe: letsencrypt::enable
* ruby_block[http external-url] action run (skipped due to only_if)
Recipe: <Dynamically Defined Resource>
* service[nginx] action nothing (skipped due to action :nothing)
Recipe: nginx::enable
* runit_service[nginx] action enable
* ruby_block[restart_service] action nothing (skipped due to action :nothing)
* ruby_block[restart_log_service] action nothing (skipped due to action :nothing)
* ruby_block[reload_log_service] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/nginx] action create (up to date)
* template[/opt/gitlab/sv/nginx/run] action create (up to date)
* directory[/opt/gitlab/sv/nginx/log] action create (up to date)
* directory[/opt/gitlab/sv/nginx/log/main] action create (up to date)
* template[/opt/gitlab/sv/nginx/log/run] action create (up to date)
* template[/var/log/gitlab/nginx/config] action create (up to date)
* ruby_block[verify_chown_persisted_on_nginx] action nothing (skipped due to action :nothing)
* directory[/opt/gitlab/sv/nginx/env] action create (up to date)
* ruby_block[Delete unmanaged env files for nginx service] action run (skipped due to only_if)
* template[/opt/gitlab/sv/nginx/check] action create (skipped due to only_if)
* template[/opt/gitlab/sv/nginx/finish] action create (skipped due to only_if)
* directory[/opt/gitlab/sv/nginx/control] action create (up to date)
* link[/opt/gitlab/init/nginx] action create (up to date)
* file[/opt/gitlab/sv/nginx/down] action delete (up to date)
* directory[/opt/gitlab/service] action create (up to date)
* link[/opt/gitlab/service/nginx] action create (up to date)
* ruby_block[wait for nginx service socket] action run (skipped due to not_if)
(up to date)
* execute[reload nginx] action nothing (skipped due to action :nothing)
Recipe: letsencrypt::enable
* directory[/etc/gitlab/ssl] action create (up to date)
* acme_selfsigned[git.domain.com] action create
* file[git.domain.com SSL selfsigned key] action create_if_missing (up to date)
* file[git.domain.com SSL selfsigned crt] action create_if_missing (up to date)
* file[git.domain.com SSL selfsigned chain] action create_if_missing (skipped due to not_if)
(up to date)
Recipe: letsencrypt::http_authorization
* letsencrypt_certificate[git.domain.com] action create
* acme_certificate[staging] action create
* file[git.domain.com SSL key] action create_if_missing (up to date)
* directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
* file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk] action create
- create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk
- update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk from none to a82106
--- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk 2019-11-12 10:01:10.660704356 +0000
+++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk20191112-3411-7my7ho 2019-11-12 10:01:10.660704356 +0000
@@ -1 +1,2 @@
+3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk.00vh3DyUV3RwbztyBQ1-_LInNXYRnr1hL8hkMr9GiX4
- change mode from '' to '0644'
- change owner from '' to 'root'
- change group from '' to 'root'
* file[git.domain.com SSL key] action nothing (skipped due to action :nothing)
* directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action nothing (skipped due to action :nothing)
* file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk] action nothing (skipped due to action :nothing)
* file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk] action delete
- delete file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/3VDkD2BkBu_4BuWVfBSmtvXdL3Yo7NQEJq_QH_7NCXk
* ruby_block[create certificate for git.domain.com] action run
================================================================================
Error executing action `run` on resource 'ruby_block[create certificate for git.domain.com]'
================================================================================
RuntimeError
------------
[git.domain.com] Validation failed, unable to request certificate
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb
108: ruby_block "create certificate for #{new_resource.cn}" do # ~FC014
109: block do
110: unless (all_validations.map { |authz| authz.status == 'valid' }).all?
111: fail "[#{new_resource.cn}] Validation failed, unable to request certificate"
112: end
113:
114: begin
115: newcert = acme_cert(order, new_resource.cn, mykey, new_resource.alt_names)
116: rescue Acme::Client::Error => e
117: fail "[#{new_resource.cn}] Certificate request failed: #{e.message}"
118: else
119: Chef::Resource::File.new("#{new_resource.cn} SSL new crt", run_context).tap do |f|
120: f.path new_resource.crt
121: f.owner new_resource.owner
122: f.group new_resource.group
123: f.content newcert
124: f.mode 00644
125: end.run_action :create
126: end
127: end
128: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:108:in `block in class_from_file'
ruby_block("create certificate for git.domain.com") do
action [:run]
default_guard_interpreter :default
declared_type :ruby_block
cookbook_name "letsencrypt"
block #<Proc:0x00005651ba874f30@/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:109>
block_name "create certificate for git.domain.com"
end
System Info:
------------
chef_version=14.13.11
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
================================================================================
Error executing action `create` on resource 'acme_certificate[staging]'
================================================================================
RuntimeError
------------
ruby_block[create certificate for git.domain.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.domain.com] Validation failed, unable to request certificate
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'
Resource Declaration:
---------------------
suppressed sensitive resource output
Compiled Resource:
------------------
suppressed sensitive resource output
System Info:
------------
chef_version=14.13.11
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
================================================================================
Error executing action `create` on resource 'letsencrypt_certificate[git.domain.com]'
================================================================================
RuntimeError
------------
acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: RuntimeError: ruby_block[create certificate for git.domain.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [git.domain.com] Validation failed, unable to request certificate
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb:111:in `block (3 levels) in class_from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb
5: letsencrypt_certificate site do
6: crt node['gitlab']['nginx']['ssl_certificate']
7: key node['gitlab']['nginx']['ssl_certificate_key']
8: notifies :run, "execute[reload nginx]", :immediate
9: notifies :run, 'ruby_block[display_le_message]'
10: only_if { omnibus_helper.service_up?('nginx') }
11: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:5:in `from_file'
letsencrypt_certificate("git.domain.com") do
action [:create]
updated true
updated_by_last_action true
default_guard_interpreter :default
declared_type :letsencrypt_certificate
cookbook_name "letsencrypt"
recipe_name "http_authorization"
crt "/etc/gitlab/ssl/git.domain.com.crt"
key "/etc/gitlab/ssl/git.domain.com.key"
alt_names []
cn "git.domain.com"
only_if { #code block }
end
System Info:
------------
chef_version=14.13.11
platform=ubuntu
platform_version=18.04
ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
Running handlers complete
Chef Client failed. 2 resources updated in 11 seconds
There was an error renewing Let's Encrypt certificates, please checkout the outputResults of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 12.4.2-ee Revision: a3170599aa2 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://git.domain.com HTTP Clone URL: https://git.domain.com/some-group/some-project.git SSH Clone URL: git@git.domain.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2, ultraauth
GitLab Shell Version: 10.2.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 2/2 ... yes 2/3 ... yes 2/4 ... yes 2/5 ... yes 2/6 ... yes 2/7 ... yes 2/8 ... yes 11/9 ... yes 11/10 ... yes 2/12 ... yes 12/15 ... yes 2/17 ... yes 2/19 ... yes 2/20 ... yes 13/21 ... yes 14/23 ... yes 16/24 ... yes 17/25 ... yes 17/26 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 2 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished