Skip to content

"Acme::Client::Error: JWS has invalid anti-replay nonce" error aborts SSL setup during 'gitlab-ctl reconfigure'

Summary

gitlab-ctl reconfigure throws up below errors:

Error executing action `create` on resource 'acme_certificate[staging]'
 Acme::Client::Error
      -------------------
      JWS has invalid anti-replay nonce iWsKVI6g4i3i-ZSJ9b9g8bZT6AeWtgokuo2bM2FdLy4

Error executing action `create` on resource 'letsencrypt_certificate[gitlab.mydomain.com]'
    Acme::Client::Error
    -------------------
    acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb
 line 20) had an error: Acme::Client::Error: JWS has invalid anti-replay 
 nonce iWsKVI6g4i3i-ZSJ9b9g8bZT6AeWtgokuo2bM2FdLy4

There was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[gitlab.mydomain.com] (letsencrypt::http_authorization line 3) 
had an error: Acme::Client::Error: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) 
had an error: Acme::Client::Error: JWS has invalid anti-replay nonce 
iWsKVI6g4i3i-ZSJ9b9g8bZT6AeWtgokuo2bM2FdLy4

Steps to reproduce

  1. Upgraded from 8.4.10 -> 8.17.8 -> 9.5.10 -> 10.8.5 -> 11.0.3 one by one using sudo yum install -y gitlab-ce-10.8.5, etc
  2. Everything went smooth except note1 below (that also I got it fixed). Thought of enabling letsencrypt in the final stage
  3. After upgrading to 11.0.3, couldn't find any letsencrypt related config in /etc/gitlab/gitlab.rb. But, that wasn't necessary for gitlab-ctl reconfigure
  4. GitLab works but with SSL NET::ERR_CERT_AUTHORITY_INVALID error due to above error during gitlab-ctl reconfigure
  5. After Googling, I added letsencrypt related config in /etc/gitlab/gitlab.rb. Still same error; but it presented little more information. Now, it keeps on presenting same error. Without it, it fails only once during gitlab-ctl reconfigure. Had to remove /etc/gitlab/ssl/HOSTNAME* to trigger same error.

Solutions tried

  1. Setting up letsencrypt['contact_emails']. No luck
  2. Checking if well_known URL is getting called in access log. No (It seems to fail before that)
  3. Checking system time. Correct; but in IST (e.g., Fri Jul 13 13:18:11 IST 2018)
  4. Fixing dot email (Gmail) in letsencrypt['contact_emails'], say a.b@mydomain.com to ab@mydomain.com. No luck.
  5. https://gitlab.com/gitlab-org/gitlab-ce/issues/43719#note_75864519 . No (It seems to fail before that). Reverted back
  6. Based on letsencrypt community discussions, this error may go after few tries. But, it didn't. https://community.letsencrypt.org/t/jws-has-invalid-anti-replay-nonce/30020/4
  7. Based on letsencrypt community discussions, nonce handling has to be tweaked so that it should request new nonce if there's a failure https://community.letsencrypt.org/t/jws-has-invalid-anti-replay-nonce-using-letsencryptsimple-aka-winsimple/48177/6 So, it looks like script needs to be fixed.

note1

Somewhere in the process (forgot the exact version, sorry), it suggested to fix git_data_dirs. Based, on the example it suggested, it was "path" => "/srv/gitlab/git-data" instead of mine "path" => "/git/gitlab/git-data". I blindly copy pasted and it made to show empty repositories. I had to scratch my head for 5 hours till I figure it out. I think, there are many open tickets for this issue alone. Was also getting 500 errors on few projects and this https://gitlab.com/gitlab-org/gitlab-ce/issues/21199#note_14168647 resolved that too

What is the current bug behavior?

  • GitLab instance works with SSL NET::ERR_CERT_AUTHORITY_INVALID error
  • As sudo gitlab-ctl reconfigure fails with above error

What is the expected correct behavior?

  • Green SSL certificate
  • sudo gitlab-ctl reconfigure succeeds

Relevant logs

/var/log/gitlab/reconfigure/1531466641.log 
# Logfile created on 2018-07-13 12:54:01 +0530 by logger.rb/56815
[2018-07-13T12:54:01+05:30] INFO: Started chef-zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded
  One version per cookbook

[2018-07-13T12:54:01+05:30] INFO: *** Chef 13.6.4 ***
[2018-07-13T12:54:01+05:30] INFO: Platform: x86_64-linux
[2018-07-13T12:54:01+05:30] INFO: Chef-client pid: 15299
[2018-07-13T12:54:01+05:30] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2018-07-13T12:54:05+05:30] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2018-07-13T12:54:05+05:30] INFO: Run List is [recipe[gitlab]]
[2018-07-13T12:54:05+05:30] INFO: Run List expands to [gitlab]
[2018-07-13T12:54:05+05:30] INFO: Starting Chef Run for gitlab.mydomain.com
[2018-07-13T12:54:05+05:30] INFO: Running start handlers
[2018-07-13T12:54:05+05:30] INFO: Start handlers complete.
[2018-07-13T12:54:06+05:30] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.0.0, gitaly@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@0.14.2, acme@3.1.0, crond@0.1.0, compat_resource@12.19.0]
[2018-07-13T12:54:09+05:30] WARN: Selected systemd because systemctl shows .mount units
[2018-07-13T12:54:09+05:30] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2018-07-13T12:54:10+05:30] INFO: execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] ran successfully
[2018-07-13T12:54:11+05:30] INFO: execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] ran successfully
[2018-07-13T12:54:11+05:30] INFO: execute[Guard resource] ran successfully
[2018-07-13T12:54:11+05:30] INFO: execute[Guard resource] ran successfully
[2018-07-13T12:54:20+05:30] INFO: Running queued delayed notifications before re-raising exception
[2018-07-13T12:54:20+05:30] INFO: Running queued delayed notifications before re-raising exception
[2018-07-13T12:54:20+05:30] ERROR: Running exception handlers
[2018-07-13T12:54:20+05:30] ERROR: Exception handlers complete
[2018-07-13T12:54:20+05:30] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2018-07-13T12:54:20+05:30] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-07-13T12:54:20+05:30] FATAL: Acme::Client::Error: letsencrypt_certificate[gitlab.mydomain.com] (letsencrypt::http_authorization line 3) had an error: Acme::Client::Error: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: Acme::Client::Error: JWS has invalid anti-replay nonce iWsKVI6g4i3i-ZSJ9b9g8bZT6AeWtgokuo2bM2FdLy4
/opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out 
Generated at 2018-07-13 12:54:20 +0530
Acme::Client::Error: letsencrypt_certificate[gitlab.mydomain.com] (letsencrypt::http_authorization line 3) had an error: Acme::Client::Error: acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: Acme::Client::Error: JWS has invalid anti-replay nonce iWsKVI6g4i3i-ZSJ9b9g8bZT6AeWtgokuo2bM2FdLy4
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:43:in `raise_on_error!'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:33:in `on_complete'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:18:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/response.rb:61:in `on_complete'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:18:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/rack_builder.rb:143:in `build_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/connection.rb:387:in `run_request'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/connection.rb:175:in `post'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/resources/registration.rb:20:in `agree_terms'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:40:in `acme_client'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:48:in `acme_authz_for'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:69:in `block (2 levels) in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `map'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'
(eval):2:in `block in action_create'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:211:in `instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:211:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:171:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource.rb:591:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:70:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:97:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:212:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:171:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource.rb:591:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:70:in `run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:97:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:718:in `block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:713:in `catch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:713:in `converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:752:in `converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:286:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:273:in `run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:249:in `block in run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:232:in `run_chef_client'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application/client.rb:434:in `run_application'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:59:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/bin/chef-client:26:in `'
/opt/gitlab/embedded/bin/chef-client:23:in `load'
/opt/gitlab/embedded/bin/chef-client:23:in `'









Caused by Acme::Client::Error: JWS has invalid anti-replay nonce iWsKVI6g4i3i-ZSJ9b9g8bZT6AeWtgokuo2bM2FdLy4
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:43:in raise_on_error!' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:33:in on_complete'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:18:in block in call' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/response.rb:61:in on_complete'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/faraday_middleware.rb:18:in call' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/rack_builder.rb:143:in build_response'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/connection.rb:387:in run_request' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/faraday-0.15.2/lib/faraday/connection.rb:175:in post'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/acme-client-0.4.0/lib/acme/client/resources/registration.rb:20:in agree_terms' /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:40:in acme_client'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:48:in acme_authz_for' /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:69:in block (2 levels) in class_from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in map' /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in block in class_from_file'
(eval):2:in block in action_create' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:211:in instance_eval'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:211:in compile_and_converge_action' (eval):2:in action_create'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:171:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource.rb:591:in run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:70:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in block (2 levels) in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in each' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in block in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:94:in block in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:85:in step' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:92:in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:97:in converge' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:212:in compile_and_converge_action'
(eval):2:in action_create' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/provider.rb:171:in run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource.rb:591:in run_action' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:70:in run_action'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in block (2 levels) in converge' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in each'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:98:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:94:in block in execute_each_resource'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:114:in call_iterator_block' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:85:in step'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:103:in iterate' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/stepable_iterator.rb:55:in each_with_index'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/resource_collection/resource_list.rb:92:in execute_each_resource' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/runner.rb:97:in converge'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:718:in block in converge' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:713:in catch'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:713:in converge' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:752:in converge_and_save'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/client.rb:286:in run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:273:in run_with_graceful_exit_option'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:249:in block in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/local_mode.rb:44:in with_server_connectivity'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:232:in run_chef_client' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application/client.rb:434:in run_application'
/opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/lib/chef/application.rb:59:in run' /opt/gitlab/embedded/lib/ruby/gems/2.4.0/gems/chef-13.6.4/bin/chef-client:26:in <top (required)>'
/opt/gitlab/embedded/bin/chef-client:23:in load' /opt/gitlab/embedded/bin/chef-client:23:in '
Output of `gitlab-ctl renew-le-certs` (taken after sometime, edit) 
Starting Chef Client, version 13.6.4[0m
resolving cookbooks for run list: ["gitlab::letsencrypt_renew"][0m
Synchronizing Cookbooks:[0m
  - gitlab (0.0.1)[0m
  - package (0.1.0)[0m
  - postgresql (0.1.0)[0m
  - registry (0.1.0)[0m
  - mattermost (0.1.0)[0m
  - consul (0.0.0)[0m
  - gitaly (0.1.0)[0m
  - letsencrypt (0.1.0)[0m
  - nginx (0.1.0)[0m
  - runit (0.14.2)[0m
  - acme (3.1.0)[0m
  - crond (0.1.0)[0m
  - compat_resource (12.19.0)[0m
Installing Cookbook Gems:[0m
Compiling Cookbooks...[0m
Converging 54 resources[0m
Recipe: letsencrypt::enable[0m
  * ruby_block[http external-url] action run (skipped due to only_if)
Recipe: runit::systemd[0m
  * directory[/usr/lib/systemd/system] action create (up to date)
  * cookbook_file[/usr/lib/systemd/system/gitlab-runsvdir.service] action create (up to date)
  * file[/etc/systemd/system/default.target.wants/gitlab-runsvdir.service] action delete (up to date)
  * execute[systemctl daemon-reload] action nothing (skipped due to action :nothing)
  * execute[systemctl enable gitlab-runsvdir] action nothing (skipped due to action :nothing)
  * execute[systemctl start gitlab-runsvdir] action nothing (skipped due to action :nothing)
Recipe: nginx::enable[0m
  * directory[/opt/gitlab/sv/nginx] action create (up to date)
  * directory[/opt/gitlab/sv/nginx/log] action create (up to date)
  * directory[/opt/gitlab/sv/nginx/log/main] action create (up to date)
  * template[/opt/gitlab/sv/nginx/run] action create (up to date)
  * template[/opt/gitlab/sv/nginx/log/run] action create (up to date)
  * template[/var/log/gitlab/nginx/config] action create (up to date)
  * ruby_block[reload nginx svlogd configuration] action nothing (skipped due to action :nothing)
  * ruby_block[restart nginx svlogd configuration] action nothing (skipped due to action :nothing)
  * file[/opt/gitlab/sv/nginx/down] action delete (up to date)
  * link[/opt/gitlab/init/nginx] action create (up to date)
  * link[/opt/gitlab/service/nginx] action create (up to date)
  * ruby_block[supervise_nginx_sleep] action run (skipped due to not_if)
  * directory[/opt/gitlab/sv/nginx/supervise] action create (up to date)
  * directory[/opt/gitlab/sv/nginx/log/supervise] action create (up to date)
  * file[/opt/gitlab/sv/nginx/supervise/ok] action touch (skipped due to only_if)
  * file[/opt/gitlab/sv/nginx/log/supervise/ok] action touch (skipped due to only_if)
  * file[/opt/gitlab/sv/nginx/supervise/control] action touch (skipped due to only_if)
  * file[/opt/gitlab/sv/nginx/log/supervise/control] action touch (skipped due to only_if)
  * service[nginx] action nothing (skipped due to action :nothing)
  * execute[reload nginx] action nothing (skipped due to action :nothing)
Recipe: letsencrypt::enable[0m
  * directory[/etc/gitlab/ssl] action create (up to date)
  * acme_selfsigned[gitlab.mydomain.com] action create
    * file[gitlab.mydomain.com SSL selfsigned key] action create_if_missing (up to date)
    * file[gitlab.mydomain.com SSL selfsigned crt] action create_if_missing (up to date)
    * file[gitlab.mydomain.com SSL selfsigned chain] action create_if_missing (skipped due to not_if)
     (up to date)
Recipe: letsencrypt::http_authorization[0m
  * letsencrypt_certificate[gitlab.mydomain.com] action create
    * acme_certificate[staging] action create
      * file[gitlab.mydomain.com SSL key] action create_if_missing (up to date)
      [0m
      ================================================================================[0m
      [31mError executing action `create` on resource 'acme_certificate[staging]'[0m
      ================================================================================[0m

[0m      Acme::Client::Error[0m
-------------------[0m
JWS has invalid anti-replay nonce nJbR_tPIV3IzukarwUvIWQiyeduQdJD3wPJFVNRu6kk[0m


[0m      Cookbook Trace:[0m
---------------[0m
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:39:in acme_client' [0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:48:in acme_authz_for'
[0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:69:in block (2 levels) in class_from_file' [0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in map'
[0m      /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'[0m


[0m      Resource Declaration:[0m
---------------------[0m
suppressed sensitive resource output[0m


[0m      Compiled Resource:[0m
------------------[0m
suppressed sensitive resource output[0m


[0m      System Info:[0m
------------[0m
chef_version=13.6.4
[0m      platform=centos
[0m      platform_version=7.4.1708
[0m      ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
[0m      program_name=/opt/gitlab/embedded/bin/chef-client
[0m      executable=/opt/gitlab/embedded/bin/chef-client[0m


[0m    [0m
================================================================================[0m
[31mError executing action create on resource 'letsencrypt_certificate[gitlab.mydomain.com]'[0m
================================================================================[0m


[0m    Acme::Client::Error[0m
-------------------[0m
acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 20) had an error: Acme::Client::Error: JWS has invalid anti-replay nonce nJbR_tPIV3IzukarwUvIWQiyeduQdJD3wPJFVNRu6kk[0m


[0m    Cookbook Trace:[0m
---------------[0m
/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:39:in acme_client' [0m    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/libraries/acme.rb:48:in acme_authz_for'
[0m    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:69:in block (2 levels) in class_from_file' [0m    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in map'
[0m    /opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/providers/certificate.rb:68:in `block in class_from_file'[0m


[0m    Resource Declaration:[0m
---------------------[0m
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb
[0m

[0m      3: letsencrypt_certificate site do
[0m      4:   fullchain node['gitlab']['nginx']['ssl_certificate']
[0m      5:   key node['gitlab']['nginx']['ssl_certificate_key']
[0m      6:   notifies :run, "execute[reload nginx]", :immediate
[0m      7:   notifies :run, 'ruby_block[display_le_message]'
[0m      8: end
[0m

[0m    Compiled Resource:[0m
------------------[0m
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/recipes/http_authorization.rb:3:in `from_file'
[0m

[0m    letsencrypt_certificate("gitlab.mydomain.com") do
[0m      action [:create]
[0m      default_guard_interpreter :default
[0m      declared_type :letsencrypt_certificate
[0m      cookbook_name "letsencrypt"
[0m      recipe_name "http_authorization"
[0m      fullchain "/etc/gitlab/ssl/gitlab.mydomain.com.crt"
[0m      key "/etc/gitlab/ssl/gitlab.mydomain.com.key"
[0m      alt_names []
[0m      cn "gitlab.mydomain.com"
[0m    end
[0m

[0m    System Info:[0m
------------[0m
chef_version=13.6.4
[0m    platform=centos
[0m    platform_version=7.4.1708
[0m    ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
[0m    program_name=/opt/gitlab/embedded/bin/chef-client
[0m    executable=/opt/gitlab/embedded/bin/chef-client[0m


[0m[0m
Running handlers:[0m
Running handlers complete
[0mChef Client failed. 0 resources updated in 14 seconds[0m

Details of package version

Provide the package version installation details 
gitlab-runner-10.4.0-1.x86_64
gitlab-ce-11.0.3-ce.0.el7.x86_64

Environment details

  • Operating System: CentOS Linux 7 (Core)
  • Installation Target:
    • Bare Metal Machine
  • Installation Type:
    • Upgrade from version 8.4.10 -> 8.17.8 -> 9.5.10 -> 10.8.5 -> 11.0.3
  • Is there any other software running on the machine: No
  • Is this a single or multiple node installation? Single
  • Resources
    • CPU: Intel(R) Xeon(R) CPU E5-2407 v2 @ 2.40GHz
    • Memory total: 16215536 kB

Configuration details

/etc/gitlab/gitlab.rb (Note: Markdown viewer is trimming last few lines) 
external_url 'https://gitlab.mydomain.com'
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['foo@mydomain.com'] # This should be an array of email addresses to add as contacts
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = 0
letsencrypt['auto_renew_minute'] = 0 # Should be a number or cron expression, if specified.
letsencrypt['auto_renew_day_of_month'] = "*/4"
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below
  snip
 EOS
git_data_dirs({
    "default" => {
        "path" => "/git/gitlab/git-data"
    }
})
Edited by R. Rajesh Jeba Anbiah