OpenSSL PKey error thrown when generating ID token for oauth2
Summary
We use Dex for authentication to our Kubernetes environment and use Gitlab to generate an ID token for the user. Previously we had this setup running when we were using gitlab omnibus (version 11.1.8
) however since migrating to a helm based deployment of Gitlab (version 12.1.4
) this authentication is no longer working. Looking in the logs It appears the issue is occurring when GitLab attempts to generate the token.
Steps to reproduce
- Follow the steps outlined here to add the application
- Follow the steps here to add the generated Application ID and secret to Dex to enable it to communicate to Gitlab
- User tries to generate an ID token using Dex
- Error is thrown in /var/log/gitlab/production.log inside the Unicorn container, resulting in a 500 error being returned to Dex
What is the current bug behavior?
User tries to authenticate through Dex which causes an OpenSSL::PKey::PKeyError inside Gitlab when generating the ID token
What is the expected correct behavior?
Gitlab authenticates the user and returns an ID token for that user
Relevant logs and/or screenshots
Logs from unicorn pod
==> /var/log/gitlab/production.log <==
Started POST "/oauth/token" for 100.96.17.85 at 2019-08-20 16:20:00 +0000
Processing by Doorkeeper::TokensController#create as HTML
Parameters: {"client_id"=>"123456789", "code"=>"[FILTERED]", "grant_type"=>"authorization_code", "redirect_uri"=>"https://dex.dev.identos.ca/callback", "scope"=>"read_user openid"}
Completed 500 Internal Server Error in 17ms
==> /var/log/gitlab/production_json.log <==
{"method":"
POST","path":"/oauth/token","format":"html","controller":"Doorkeeper::TokensController","action":"create","status":500,"error":"OpenSSL::PKey::PKeyError: Could not parse PKey: no start line","duration":16.96,"view":0.0,"time":"2019-08-20T16:20:00.542Z","params":[{"key":"client_id","value":"123456789"},{"key":"code","value":"[FILTERED]"},{"key":"grant_type","value":"authorization_code"},{"key":"redirect_uri","value":"https://dex.dev.identos.ca/callback"},{"key":"scope","value":"read_user openid"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"123456789"}
==> /var/log/gitlab/production.log <==
OpenSSL::PKey::PKeyError (Could not parse PKey: no start line):
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Helm chart deployment values
edition: ce
gitlabVersion: 12.1.4
hosts:
domain: identos.ca
hostSuffix: git
https: true
gitlab:
name: git.identos.ca
https: true
registry:
name: registry.git.identos.ca
smtp:
enabled: true
address: "email-smtp.us-east-1.amazonaws.com"
port: 2587
user_name: REDACTED
password:
secret: gitlab-smtp-password
key: password
domain: "git.identos.ca"
authentication: "login"
starttls_auto: true
openssl_verify_mode: "none"
ingress:
enabled: true
configureCertmanager: false
class: nginx
annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
certmanager.k8s.io/acme-challenge-type: http01
tls:
enabled: true
secretName: gitlab-cert
minio:
enabled: false
registry:
bucket: identos-gitlab-registry
appConfig:
omniauth:
enabled: true
allowSingleSignOn: ['google_oauth2']
blockAutoCreatedUsers: false
providers:
- secret: gitlab-google-oauth2
lfs:
bucket: identos-gitlab-lfs-objects
connection: # https://gitlab.com/charts/gitlab/blob/master/doc/charts/globals.md#connection
secret: gitlab-s3-connection
key: connection
artifacts:
bucket: identos-gitlab-artifacts
connection:
secret: gitlab-s3-connection
key: connection
uploads:
bucket: identos-gitlab-uploads
connection:
secret: gitlab-s3-connection
key: connection
packages:
bucket: identos-gitlab-packages
connection:
secret: gitlab-s3-connection
key: connection
registry:
bucket: identos-gitlab-registry
backups:
bucket: identos-gitlab-backup
tmpBucket: identos-gitlab-backup-tmp
registry:
enabled: true
ingress:
enabled: true
tls:
secretName: gitlab-registry-cert
storage:
secret: gitlab-registry-s3
key: config
gitlab:
gitaly:
persistence:
enabled: true
storageClass: standard
accessMode: ReadWriteOnce
size: 50Gi
matchLabels: {}
matchExpressions: []
task-runner:
backups:
cron:
enabled: true
schedule: "0 21 * * *"
extraArgs: "--skip registry"
objectStorage:
config:
secret: gitlab-backup-config
key: config
unicorn:
ingress:
tls:
secretName: gitlab-cert
gitlab-runner:
install: false
nginx-ingress:
enabled: false
certmanager:
install: false
prometheus:
install: false
Possible fixes
Based on the log outputs I see, it looks like the issue is inside Doorkeeper::TokensController#create