Let's Encrypt bug: can't validate certificate
Summary
Using integrated Let's Encrypt in a working Omnibus install, straight from https://docs.gitlab.com/omnibus/settings/ssl.html#enabling doesn't work: Validation failed for domain gitlab.example.com
(my real domain name has been replaced with gitlab.example.com)
Steps to reproduce
Use a public URL like https://gitlab.example.com
, verify that the machine it runs on is publicly accessible on ports 80 & 443, enable letsencrypt support, and run gitlab-ctl reconfigure
What is the current bug behavior?
gitlab-ctl reconfigure fails with:
Recipe: letsencrypt::http_authorization
* letsencrypt_certificate[gitlab.example.com] action create
* acme_certificate[staging] action create
* file[gitlab.example.com SSL key] action create_if_missing (up to date)
* directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
* file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] action create
- create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk
- update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk from none to 4157a4
--- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk 2018-03-13 15:48:33.773681941 +0100
+++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk20180313-27454-1fw6b9p 2018-03-13 15:48:33.772681868 +0100
@@ -1 +1,2 @@
+6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk.xzONx57pyp6NmR45_sLYpCgd7BcSjTt-8u8xtVK0kZk
- change mode from '' to '0644'
- change owner from '' to 'root'
- change group from '' to 'root'
- restore selinux security context
================================================================================
Error executing action `create` on resource 'acme_certificate[staging]'
================================================================================
RuntimeError
------------
[gitlab.example.com] Validation failed for domain gitlab.example.com
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:93:in `block (2 levels) in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'
What is the expected correct behavior?
A successful reconfiguration step, and an https-accessible self-hosted gitlab server.
Relevant logs
Relevant logs
# Logfile created on 2018-03-13 15:46:57 +0100 by logger.rb/56438 [2018-03-13T15:46:57+01:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /opt/gitlab/embedded One version per cookbook[2018-03-13T15:46:57+01:00] INFO: Forking chef instance to converge... [2018-03-13T15:46:57+01:00] INFO: *** Chef 12.21.31 *** [2018-03-13T15:46:57+01:00] INFO: Platform: x86_64-linux [2018-03-13T15:46:57+01:00] INFO: Chef-client pid: 27454 [2018-03-13T15:46:57+01:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping... [2018-03-13T15:47:01+01:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/nodes/gtscpt1 [2018-03-13T15:47:01+01:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options [2018-03-13T15:47:02+01:00] INFO: Run List is [recipe[gitlab]] [2018-03-13T15:47:02+01:00] INFO: Run List expands to [gitlab] [2018-03-13T15:47:02+01:00] INFO: Starting Chef Run for gtscpt1 [2018-03-13T15:47:02+01:00] INFO: Running start handlers [2018-03-13T15:47:02+01:00] INFO: Start handlers complete. [2018-03-13T15:47:02+01:00] INFO: HTTP Request Returned 404 Not Found: Object not found: [2018-03-13T15:47:03+01:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.0.0, gitaly@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@0.14.2, acme@3.1.0, compat_resource@12.19.0] [2018-03-13T15:47:09+01:00] WARN: Selected systemd because systemctl shows .mount units [2018-03-13T15:47:10+01:00] INFO: HTTP Request Returned 404 Not Found: Object not found: [2018-03-13T15:47:10+01:00] INFO: execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] ran successfully [2018-03-13T15:47:28+01:00] INFO: bash[Set proper security context on ssh files for selinux] ran successfully [2018-03-13T15:47:28+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/etc/gitlab.yml.chef-20180313154728.812076 [2018-03-13T15:47:28+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/etc/gitlab.yml.chef-20180222221355.172007 [2018-03-13T15:47:28+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/gitlab.yml [2018-03-13T15:47:29+01:00] INFO: execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] ran successfully [2018-03-13T15:47:33+01:00] INFO: execute[Guard resource] ran successfully [2018-03-13T15:47:38+01:00] INFO: execute[Guard resource] ran successfully [2018-03-13T15:47:39+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf/gitlab-http.conf.chef-20180313154739.240874 [2018-03-13T15:47:39+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf/gitlab-http.conf.chef-20180223102545.634755 [2018-03-13T15:47:39+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] updated file contents /var/opt/gitlab/nginx/conf/gitlab-http.conf [2018-03-13T15:47:39+01:00] INFO: template[/opt/gitlab/etc/gitlab-healthcheck-rc] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/opt/gitlab/etc/gitlab-healthcheck-rc.chef-20180313154739.498880 [2018-03-13T15:47:39+01:00] INFO: template[/opt/gitlab/etc/gitlab-healthcheck-rc] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/opt/gitlab/etc/gitlab-healthcheck-rc.chef-20171003103928.981560 [2018-03-13T15:47:39+01:00] INFO: template[/opt/gitlab/etc/gitlab-healthcheck-rc] updated file contents /opt/gitlab/etc/gitlab-healthcheck-rc [2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] created file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk [2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] updated file contents /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk [2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] owner changed to 0 [2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] group changed to 0 [2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] mode changed to 644 [2018-03-13T15:48:48+01:00] INFO: Running queued delayed notifications before re-raising exception [2018-03-13T15:48:48+01:00] INFO: Running queued delayed notifications before re-raising exception [2018-03-13T15:48:48+01:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed) [2018-03-13T15:49:08+01:00] INFO: execute[clear the gitlab-rails cache] ran successfully [2018-03-13T15:49:08+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] sending restart action to service[nginx] (delayed) [2018-03-13T15:49:08+01:00] INFO: service[nginx] restarted [2018-03-13T15:49:08+01:00] ERROR: Running exception handlers [2018-03-13T15:49:08+01:00] ERROR: Exception handlers complete [2018-03-13T15:49:08+01:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out [2018-03-13T15:49:08+01:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report [2018-03-13T15:49:08+01:00] ERROR: letsencrypt_certificate[gitlab.example.com] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.example.com] Validation failed for domain gitlab.example.com [2018-03-13T15:49:08+01:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Checking GitLab Shell ... GitLab Shell version >= 6.0.3 ? ... OK (6.0.3) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 13/1 ... ok 13/2 ... ok 13/3 ... ok 13/4 ... ok 13/5 ... ok 13/6 ... ok 13/7 ... ok 13/8 ... ok 13/9 ... ok 13/10 ... ok 13/12 ... ok 13/13 ... ok 13/14 ... ok 6/15 ... ok 13/16 ... ok 13/18 ... ok 13/20 ... ok 13/21 ... ok 13/23 ... ok 13/25 ... ok 17/26 ... ok 13/27 ... ok 13/28 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Reply by email is disabled in config/gitlab.yml Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 13/1 ... yes 13/2 ... yes 13/3 ... yes 13/4 ... yes 13/5 ... yes 13/6 ... yes 13/7 ... yes 13/8 ... yes 13/9 ... yes 13/10 ... yes 13/12 ... yes 13/13 ... yes 13/14 ... yes 6/15 ... yes 13/16 ... yes 13/18 ... yes 13/20 ... yes 13/21 ... yes 13/23 ... yes 13/25 ... yes 17/26 ... yes 13/27 ... yes 13/28 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 11 Checking GitLab ... Finished
Details of package version
Provide the package version installation details
gitlab-runner-10.5.0-1.x86_64 gitlab-ce-10.5.4-ce.0.el7.x86_64
System information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknownGitLab information Version: 10.5.4 Revision: 8d768f9 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://gitlab.example.com HTTP Clone URL: http://gitlab.example.com/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 6.0.3 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Environment details
- Operating System:
CentOS Linux release 7.4.1708 (Core)
- Installation Target:
- Bare Metal Machine
- Installation Type:
- Upgrade from version
10.4
- Upgrade from version
- Is there any other software running on the machine:
SMB server
- This is a single node installation
- Resources
- CPU:
4 x Intel(R) Xeon(R) CPU E5-2403
- Memory total:
8Gb
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://gitlab.example.com'gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_email_from'] = 'tools@example.com' gitlab_rails['gitlab_email_display_name'] = 'Gitlab example' gitlab_rails['gitlab_email_reply_to'] = 'tools@example.com' gitlab_rails['gitlab_email_subject_suffix'] = '' gitlab_rails['gitlab_default_theme'] = 3
gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "smtp.office365.com" gitlab_rails['smtp_port'] = 587 gitlab_rails['smtp_user_name'] = "tools@example.com" gitlab_rails['smtp_password'] = "x" gitlab_rails['smtp_domain'] = "example.com" gitlab_rails['smtp_authentication'] = "login" gitlab_rails['smtp_enable_starttls_auto'] = true gitlab_rails['smtp_tls'] = false
letsencrypt['enable'] = true letsencrypt['contact_emails'] = ['xav@example.com']