Unable to renew letsencrypt certificate anymore
Summary
I can't renew letsencrypt certificate anymore while it was working fine for several months. I'm running Omnibus, starter edition.
Steps to reproduce
I'm running omnibus on my own company server, I was using GitLab 12.5.0 and I wanted to upgrade to 12.5.3 when I figured out this issue.
As suggested by the error message during the upgrade attempt, I ran sudo gitlab-ctl reconfigure
to try to fix the issue but then I had this letsencrypt error message.
Example Project
Not sure I can provide anything to help here...
What is the current bug behavior?
When running gitlab-ctl reconfigure
or gitlab-ctl renew-le-certs
, I'm having the following error:
Acme::Client::Error::Malformed
------------------------------
acme_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 25) had an error: Acme::Client::Error::Malformed: Method not allowed
The full log is attached: sudo_gitlab-ctl_renew-le-certs.log
Please note that:
- the tcp port 80 and 443 are open to the internet (tested with telnet)
- I have tried to shutdown the firewall of the host running gitlab: no change
- this worked fine during several months
- I've run through a lot of similar issues but no one had the exact same error message and all the working fixes (mainly about not open ports and wrong gitlab.rb configuration...) didn't work on my side
- my gitlab.rb contains the following:
# nginx['enable'] = true
# nginx['client_max_body_size'] = '250m'
nginx['redirect_http_to_https'] = true
nginx['redirect_http_to_https_port'] = 80
...
letsencrypt['enable'] = true
# letsencrypt['contact_emails'] = # This should be an array of email addresses to add as contacts
# letsencrypt['group'] = 'root'
# letsencrypt['key_size'] = 2048
# letsencrypt['owner'] = 'root'
# letsencrypt['wwwroot'] = '/var/opt/gitlab/nginx/www'
# See http://docs.gitlab.com/omnibus/settings/ssl.html#automatic-renewal for more on these sesttings
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = "2"
# letsencrypt['auto_renew_minute'] = nil # Should be a number or cron expression, if specified.
letsencrypt['auto_renew_day_of_month'] = "*/20"
What is the expected correct behavior?
gitlab-ctl reconfigure
should run successfully...
Relevant logs and/or screenshots
see above.
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.5.3-ee Revision: 63955893276 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://gitlab-server-name.company.com HTTP Clone URL: https://gitlab-server-name.company.com/some-group/some-project.git SSH Clone URL: git@gitlab-server-name.company.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 10.2.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 7 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 19/1 ... yes 3/4 ... yes 3/5 ... yes 7/10 ... yes 3/11 ... yes 3/13 ... yes 3/14 ... yes 3/15 ... yes 2/17 ... yes 13/19 ... yes 3/21 ... yes 17/23 ... yes 2/24 ... yes 19/25 ... yes 19/26 ... yes 19/28 ... yes 19/29 ... yes 21/30 ... yes 21/31 ... yes 21/32 ... yes 21/33 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 8 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)