OpenIDConnect custom-ca for provider
Summary
OpenID connect fails with OpenIDConnect::Discovery::DiscoveryFailed (SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)):
Steps to reproduce
- Deploy the k8s (used kubespray, should work with others)
- Deploy metallb loadbalancer, configure nfs storage class.
- Deploy nginx-ingress and cert-manager, configure cluster issuer with self signed CA certificate
- Deploy keycloak with https ingress, configure
kubernetes
realm - Add same CA certificate as a
cluster-ca
secret in gitlab namespace, create provider secret (see below) - Deploy gitlab
- Click Keycloak single-sign-on button
Configuration used
global:
application:
create: false
ingress:
configureCertmanager: false
annotations:
kubernetes.io/ingress.provider: "nginx"
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/force-ssl-redirect: "true"
certmanager.k8s.io/cluster-issuer: "cluster-ca"
enabled: true
tls:
enabled: true
omniauth:
enabled: true
autoSignInWithProvider:
syncProfileFromProvider: ['keycloak']
syncProfileAttributes: ['email']
allowSingleSignOn: ['keycloak']
blockAutoCreatedUsers: false
autoLinkLdapUser: false
autoLinkSamlUser: false
externalProviders: []
providers:
- secret: gitlab-keycloak-provider
certificates:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates
tag: 20171114-r3
customCAs:
- secret: cluster-ca # same as used for ingress
certmanager:
install: false
nginx-ingress:
enabled: false
# gitlab-keycloak-provider.yml
name: 'openid_connect'
label: 'OIDC'
args:
name: 'keycloak'
scope: ['email', 'openid', 'profile']
response_type: 'code'
issuer: 'https://keycloak.{{ domain_name }}'
discovery: true
client_auth_method: 'query'
client_options:
identifier: '{{ client_id }}'
secret: '{{ kube_oidc_client_secret }}'
redirect_uri: 'https://gitlab.{{ domain_name }}/users/auth/keycloak/callback'
Current behavior
unicorn pods doesn't trust the certificate.
Expected behavior
Should allow self signed root certificate to be used for authentication.
Versions
- Chart: 7b7d95df819c61d21c8b27548cfd95716754d021
- Platform:
- Self-hosted: kubespray
- Kubernetes: (
kubectl version
)- Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
- Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (
helm version
)- Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
- Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Relevant logs
OpenIDConnect::Discovery::DiscoveryFailed (SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)):
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Additional info
In gitlab-unicorn
pods.
Originally, openssl s_client -servername keycloak.{{ domain_name }} -connect keycloak.{{ domain_name }}:443 </dev/null
from the container failed to verify keycloak certificate.
Can fix openssl with following commands executed when directly connecting to the containers
c_rehash -v /usr/lib/ssl/certs
c_rehash -v /etc/ssl/certs/
Fixes the openssl issue, unicorn still doesn't pickup the certificate.
Additionally tried explicitly setting environment variable SSL_CERT_FILE to '/usr/lib/ssl/certs/ca-certificates.crt' for pods with no luck.