Cert-manager Helm Deployment fails with message: Your ACME client is too old

Summary

Edit: Verison GitLab FOSS 12.2.1

Last Friday, Novemeber 29th, 2019, I deployed a new K8s cluster and installed cert-manager via the K8s integration within GitLab. I noticed that SSL support wasn't working for any deployed projects and Ingress was using the Fake ingress certs that are generated.

Upon further investigation into the logs of the cert-manger pod I found this little gem

I1204 19:17:43.795952       1 sync.go:120] Issuer letsencrypt-prod not ready

Which led me to find out about the ClusterIssuer. Describing it reveals a status message that doesn't look too promising, Your ACME client is too old. I've included the log below for the ClusterIssuer.

I know Let's Encrypt has started blocking old versions to disable ACMEv1, which seems to fit with the issue I'm having. After looking through #1568 I thought I was onto something with the Helm chart being out of date. However, as is pointed out in that issue, the client is already using ACMEv2 even though the version number is older.

The other cluster I have is working fine, but it was deployed back in the summer. It seems that the client that was freshly deployed is out of date.

One thing I can't find is where the ClusterIssuer gets the messages from, I'm assuming the acme client is running somewhere, so I'm unsure of what version I'm using.

Any help would be appreciated.

Info about my ClusterIssuer,

$ kubectl describe clusterissuer letsencrypt-prod
Name:         letsencrypt-prod
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2019-11-29T22:24:03Z
  Generation:          2
  Resource Version:    8799
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-prod
  UID:                 f2c0a2dd-12f6-11ea-a499-redacted
Spec:
  Acme:
    Email:  redacted@redacted.ca
    http01:
    Private Key Secret Ref:
      Key:
      Name:  letsencrypt-prod
    Server:  https://acme-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Uri:
  Conditions:
    Last Transition Time:  2019-11-29T22:24:11Z
    Message:               Failed to verify ACME account: acme: urn:ietf:params:acme:error:rateLimited: Your ACME client is too old. Please upgrade to a newer version.
    Reason:                ErrRegisterACMEAccount
    Status:                False
    Type:                  Ready
Events:                    <none>

Steps to reproduce

  1. Setup a new K8s cluster on GKE.
  2. Deploy Helm Tiller and then install Ingress and cert-manager.
  3. Deploy a project to the cluster.

What is the current bug behavior?

The cert-manager runs into an issue trying to register with the ACMEv2 server. Which causes certificates to fail to be issued.

What is the expected correct behavior?

The cert-manager should register with the API and issue certificates.

Edit: Forgot to mention the version we are using.

Edited by Scott Morris