Skip to content

OpenIDConnect custom-ca for provider

Summary

OpenID connect fails with OpenIDConnect::Discovery::DiscoveryFailed (SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)):

Steps to reproduce

  1. Deploy the k8s (used kubespray, should work with others)
  2. Deploy metallb loadbalancer, configure nfs storage class.
  3. Deploy nginx-ingress and cert-manager, configure cluster issuer with self signed CA certificate
  4. Deploy keycloak with https ingress, configure kubernetes realm
  5. Add same CA certificate as a cluster-ca secret in gitlab namespace, create provider secret (see below)
  6. Deploy gitlab
  7. Click Keycloak single-sign-on button

Configuration used

global:
  application:
    create: false
  ingress:
    configureCertmanager: false
    annotations:
      kubernetes.io/ingress.provider: "nginx"
      kubernetes.io/ingress.class: "nginx"
      ingress.kubernetes.io/force-ssl-redirect: "true"
      certmanager.k8s.io/cluster-issuer: "cluster-ca"
    enabled: true
    tls:
      enabled: true
    omniauth:
      enabled: true
      autoSignInWithProvider:
      syncProfileFromProvider: ['keycloak']
      syncProfileAttributes: ['email']
      allowSingleSignOn: ['keycloak']
      blockAutoCreatedUsers: false
      autoLinkLdapUser: false
      autoLinkSamlUser: false
      externalProviders: []
      providers:
      - secret: gitlab-keycloak-provider
  certificates:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates
      tag: 20171114-r3
    customCAs: 
    - secret: cluster-ca     # same as used for ingress
certmanager:
  install: false
nginx-ingress:
  enabled: false


# gitlab-keycloak-provider.yml
name: 'openid_connect'
label: 'OIDC'
args:
  name: 'keycloak'
  scope: ['email', 'openid', 'profile']
  response_type: 'code'
  issuer: 'https://keycloak.{{ domain_name }}'
  discovery: true
  client_auth_method: 'query'
  client_options:
    identifier: '{{ client_id }}'
    secret: '{{ kube_oidc_client_secret }}'
    redirect_uri: 'https://gitlab.{{ domain_name }}/users/auth/keycloak/callback'

Current behavior

unicorn pods doesn't trust the certificate.

Expected behavior

Should allow self signed root certificate to be used for authentication.

Versions

  • Chart: 7b7d95df819c61d21c8b27548cfd95716754d021
  • Platform:
    • Self-hosted: kubespray
  • Kubernetes: (kubectl version)
    • Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
    • Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version)
    • Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
    • Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

Relevant logs

OpenIDConnect::Discovery::DiscoveryFailed (SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)):

lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Additional info

In gitlab-unicorn pods.

Originally, openssl s_client -servername keycloak.{{ domain_name }} -connect keycloak.{{ domain_name }}:443 </dev/null from the container failed to verify keycloak certificate.

Can fix openssl with following commands executed when directly connecting to the containers

c_rehash -v /usr/lib/ssl/certs
c_rehash -v /etc/ssl/certs/

Fixes the openssl issue, unicorn still doesn't pickup the certificate.

Additionally tried explicitly setting environment variable SSL_CERT_FILE to '/usr/lib/ssl/certs/ca-certificates.crt' for pods with no luck.