Let's Encrypt bug: can't validate certificate (#3279) · Issues · GitLab.org / omnibus-gitlab

Let's Encrypt bug: can't validate certificate

Summary

Using integrated Let's Encrypt in a working Omnibus install, straight from https://docs.gitlab.com/omnibus/settings/ssl.html#enabling doesn't work: Validation failed for domain gitlab.example.com

(my real domain name has been replaced with gitlab.example.com)

Steps to reproduce

Use a public URL like https://gitlab.example.com, verify that the machine it runs on is publicly accessible on ports 80 & 443, enable letsencrypt support, and run gitlab-ctl reconfigure

What is the current bug behavior?

gitlab-ctl reconfigure fails with:
Recipe: letsencrypt::http_authorization
  * letsencrypt_certificate[gitlab.example.com] action create
    * acme_certificate[staging] action create
      * file[gitlab.example.com SSL key] action create_if_missing (up to date)
      * directory[/var/opt/gitlab/nginx/www/.well-known/acme-challenge] action create (up to date)
      * file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] action create
        - create new file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk
        - update content in file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk from none to 4157a4
        --- /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk	2018-03-13 15:48:33.773681941 +0100
        +++ /var/opt/gitlab/nginx/www/.well-known/acme-challenge/.chef-6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk20180313-27454-1fw6b9p	2018-03-13 15:48:33.772681868 +0100
        @@ -1 +1,2 @@
        +6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk.xzONx57pyp6NmR45_sLYpCgd7BcSjTt-8u8xtVK0kZk
        - change mode from '' to '0644'
        - change owner from '' to 'root'
        - change group from '' to 'root'
        - restore selinux security context
      
      ================================================================================
      Error executing action `create` on resource 'acme_certificate[staging]'
      ================================================================================
      
      RuntimeError
      ------------
      [gitlab.example.com] Validation failed for domain gitlab.example.com
      
      Cookbook Trace:
      ---------------
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:93:in `block (2 levels) in class_from_file'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'

What is the expected correct behavior?

A successful reconfiguration step, and an https-accessible self-hosted gitlab server.

Relevant logs

# Logfile created on 2018-03-13 15:46:57 +0100 by logger.rb/56438
[2018-03-13T15:46:57+01:00] INFO: Started chef-zero at chefzero://localhost:8889 with repository at /opt/gitlab/embedded
  One version per cookbook
[2018-03-13T15:46:57+01:00] INFO: Forking chef instance to converge...
[2018-03-13T15:46:57+01:00] INFO: *** Chef 12.21.31 ***
[2018-03-13T15:46:57+01:00] INFO: Platform: x86_64-linux
[2018-03-13T15:46:57+01:00] INFO: Chef-client pid: 27454
[2018-03-13T15:46:57+01:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2018-03-13T15:47:01+01:00] INFO: HTTP Request Returned 404 Not Found: Object not found: chefzero://localhost:8889/nodes/gtscpt1
[2018-03-13T15:47:01+01:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2018-03-13T15:47:02+01:00] INFO: Run List is [recipe[gitlab]]
[2018-03-13T15:47:02+01:00] INFO: Run List expands to [gitlab]
[2018-03-13T15:47:02+01:00] INFO: Starting Chef Run for gtscpt1
[2018-03-13T15:47:02+01:00] INFO: Running start handlers
[2018-03-13T15:47:02+01:00] INFO: Start handlers complete.
[2018-03-13T15:47:02+01:00] INFO: HTTP Request Returned 404 Not Found: Object not found:
[2018-03-13T15:47:03+01:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.0.0, gitaly@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@0.14.2, acme@3.1.0, compat_resource@12.19.0]
[2018-03-13T15:47:09+01:00] WARN: Selected systemd because systemctl shows .mount units
[2018-03-13T15:47:10+01:00] INFO: HTTP Request Returned 404 Not Found: Object not found:
[2018-03-13T15:47:10+01:00] INFO: execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] ran successfully
[2018-03-13T15:47:28+01:00] INFO: bash[Set proper security context on ssh files for selinux] ran successfully
[2018-03-13T15:47:28+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/etc/gitlab.yml.chef-20180313154728.812076
[2018-03-13T15:47:28+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/etc/gitlab.yml.chef-20180222221355.172007
[2018-03-13T15:47:28+01:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] updated file contents /var/opt/gitlab/gitlab-rails/etc/gitlab.yml
[2018-03-13T15:47:29+01:00] INFO: execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] ran successfully
[2018-03-13T15:47:33+01:00] INFO: execute[Guard resource] ran successfully
[2018-03-13T15:47:38+01:00] INFO: execute[Guard resource] ran successfully
[2018-03-13T15:47:39+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf/gitlab-http.conf.chef-20180313154739.240874
[2018-03-13T15:47:39+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/nginx/conf/gitlab-http.conf.chef-20180223102545.634755
[2018-03-13T15:47:39+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] updated file contents /var/opt/gitlab/nginx/conf/gitlab-http.conf
[2018-03-13T15:47:39+01:00] INFO: template[/opt/gitlab/etc/gitlab-healthcheck-rc] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/opt/gitlab/etc/gitlab-healthcheck-rc.chef-20180313154739.498880
[2018-03-13T15:47:39+01:00] INFO: template[/opt/gitlab/etc/gitlab-healthcheck-rc] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/opt/gitlab/etc/gitlab-healthcheck-rc.chef-20171003103928.981560
[2018-03-13T15:47:39+01:00] INFO: template[/opt/gitlab/etc/gitlab-healthcheck-rc] updated file contents /opt/gitlab/etc/gitlab-healthcheck-rc
[2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] created file /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk
[2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] updated file contents /var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk
[2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] owner changed to 0
[2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] group changed to 0
[2018-03-13T15:48:33+01:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/6JBijASy0fInS-wdu98z3NpMUnNWpydMa8fx7xviONk] mode changed to 644
[2018-03-13T15:48:48+01:00] INFO: Running queued delayed notifications before re-raising exception
[2018-03-13T15:48:48+01:00] INFO: Running queued delayed notifications before re-raising exception
[2018-03-13T15:48:48+01:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed)
[2018-03-13T15:49:08+01:00] INFO: execute[clear the gitlab-rails cache] ran successfully
[2018-03-13T15:49:08+01:00] INFO: template[/var/opt/gitlab/nginx/conf/gitlab-http.conf] sending restart action to service[nginx] (delayed)
[2018-03-13T15:49:08+01:00] INFO: service[nginx] restarted
[2018-03-13T15:49:08+01:00] ERROR: Running exception handlers
[2018-03-13T15:49:08+01:00] ERROR: Exception handlers complete
[2018-03-13T15:49:08+01:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2018-03-13T15:49:08+01:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-03-13T15:49:08+01:00] ERROR: letsencrypt_certificate[gitlab.example.com] (letsencrypt::http_authorization line 3) had an error: RuntimeError: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: RuntimeError: [gitlab.example.com] Validation failed for domain gitlab.example.com
[2018-03-13T15:49:08+01:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Checking GitLab Shell ...

GitLab Shell version >= 6.0.3 ? ... OK (6.0.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
13/1 ... ok
13/2 ... ok
13/3 ... ok
13/4 ... ok
13/5 ... ok
13/6 ... ok
13/7 ... ok
13/8 ... ok
13/9 ... ok
13/10 ... ok
13/12 ... ok
13/13 ... ok
13/14 ... ok
6/15 ... ok
13/16 ... ok
13/18 ... ok
13/20 ... ok
13/21 ... ok
13/23 ... ok
13/25 ... ok
17/26 ... ok
13/27 ... ok
13/28 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml
Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
13/1 ... yes
13/2 ... yes
13/3 ... yes
13/4 ... yes
13/5 ... yes
13/6 ... yes
13/7 ... yes
13/8 ... yes
13/9 ... yes
13/10 ... yes
13/12 ... yes
13/13 ... yes
13/14 ... yes
6/15 ... yes
13/16 ... yes
13/18 ... yes
13/20 ... yes
13/21 ... yes
13/23 ... yes
13/25 ... yes
17/26 ... yes
13/27 ... yes
13/28 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.6)
Git version >= 2.9.5 ? ... yes (2.14.3)
Git user has default SSH configuration? ... yes
Active users: ... 11

Checking GitLab ... Finished

Details of package version

Provide the package version installation details

gitlab-runner-10.5.0-1.x86_64
gitlab-ce-10.5.4-ce.0.el7.x86_64

System information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown

GitLab information Version: 10.5.4 Revision: 8d768f9 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://gitlab.example.com HTTP Clone URL: http://gitlab.example.com/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Using LDAP: no Using Omniauth: no

GitLab Shell Version: 6.0.3 Repository storage paths:

default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Environment details

Operating System: CentOS Linux release 7.4.1708 (Core)
Installation Target:
- Bare Metal Machine
Installation Type:
- Upgrade from version 10.4
Is there any other software running on the machine: SMB server
This is a single node installation
Resources
- CPU: 4 x Intel(R) Xeon(R) CPU E5-2403
- Memory total: 8Gb

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

external_url 'https://gitlab.example.com'
gitlab_rails['gitlab_email_enabled'] = true
gitlab_rails['gitlab_email_from'] = 'tools@example.com'
gitlab_rails['gitlab_email_display_name'] = 'Gitlab example'
gitlab_rails['gitlab_email_reply_to'] = 'tools@example.com'
gitlab_rails['gitlab_email_subject_suffix'] = ''
gitlab_rails['gitlab_default_theme'] = 3
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.office365.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "tools@example.com"
gitlab_rails['smtp_password'] = "x"
gitlab_rails['smtp_domain'] = "example.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['xav@example.com']

Edited Mar 15, 2018 by Xavier Bestel