Generic OAuth2 provider using Helm for k8s deployment error
Summary
Unable to configure and use a generic oauth2 (fusionauth) in a kubernetes environment using helm to deploy gitlab
Steps to reproduce
a fresh helm install was performed without the omniauth configuration in the globals
$ helm install --name=gitlab-ee --namespace=gitlab-ee gitlab/gitlab -f values.yaml
all pods are up and running correctly.
Omniauth config / helm upgrade
First, create the secret with the app id and secret created generated from fusionauth:
$ kubectl create secret generic -n gitlab-ee oauth2-generic --from-file=provider=provider.yaml
Next, the helm upgrade with the values file with the omniauth config
$ helm upgrade gitlab-ee gitlab/gitlab -f values.yaml
Configuration used for provider.yaml
name: oauth2-generic
label: Fusionauth
app_id: 'xxxxx'
app_secret: 'yyyyy'
args:
approval_prompt: ''Configuration used in the global.appConfig.omniauth
omniauth:
enabled: true
autoSignInWithProvider:
syncProfileFromProvider: []
syncProfileAttributes: ['email']
allowSingleSignOn: ['omniauth-oauth2-generic']
blockAutoCreatedUsers: true
autoLinkLdapUser: false
autoLinkSamlUser: false
externalProviders: []
allowBypassTwoFactor: []
providers:
- secret: oauth2-generic Current behavior
The helm upgrade goes through, but several pods have init errors that result in crashloopbackoffs
- gitlab-ee-gitlab-runner
- gitlab-ee-sidekiq-all-in-1-v1
- gitlab-ee-unicorn
the omniauth config does not go through, all documentation references ruby/ruby gem files; not for the kubernetes config.
Expected behavior
The pods will be upgraded and be able to use the generic oauth2 provider (fusionauth) for sso into gitlab-ee on k8s.
Versions
- Chart: gitlab-3.2.3 app: 12.9.3
- Platform:
- Self-hosted
- Kubernetes: (
kubectl version)- Client: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3"}
- Server: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11"}
- Helm: (
helm version)- Client: &version.Version{SemVer:"v2.16.1"}
- Server: &version.Version{SemVer:"v2.16.1"}
Relevant logs
kubectl describe pod -n gitlab-ee gitlab-unicorn-zzzzz after the helm upgrade returns:
Name: gitlab-ee-unicorn-68cbdbc8fc-bhgr5
Namespace: gitlab-ee
Priority: 0
Node: ip-xx.xxx.xxx.xxx
Start Time: Thu, 16 Apr 2020 12:09:07 -0700
Labels: app=unicorn
pod-template-hash=68cbdbc8fc
release=gitlab-ee
Annotations: checksum/config: add745fa2fee11eeaa29c5c0d232264bc9e515ecdc406ded094c2def30de8b67
cluster-autoscaler.kubernetes.io/safe-to-evict: true
prometheus.io/path: /-/metrics
prometheus.io/port: 8080
prometheus.io/scrape: true
Status: Pending
IP: xxx.xxx.xxx.xxx
IPs: <none>
Controlled By: ReplicaSet/gitlab-ee-unicorn-qqqqq
Init Containers:
certificates:
Container ID: docker://b13526435e35a3ebc90342491490c714463b4484403e29cfa0d0bef304740c9f
Image: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/build/cng/alpine-certificates@sha256:00ce9a585179e6b22c9bfea9ba82552630eab0bd25da4f13282b588b2ad022dc
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 16 Apr 2020 12:09:08 -0700
Finished: Thu, 16 Apr 2020 12:09:09 -0700
Ready: True
Restart Count: 0
Requests:
cpu: 50m
Environment: <none>
Mounts:
/etc/ssl/certs from etc-ssl-certs (rw)
/usr/local/share/ca-certificates from custom-ca-certificates (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nh99g (ro)
configure:
Container ID: docker://c60d5218e2d76771008fa067f8d270eeaf45d9da5c83729b07be6f251c9b86a2
Image: busybox:latest
Image ID: docker-pullable://busybox@sha256:89b54451a47954c0422d873d438509dae87d478f1cb5d67fb130072f67ca5d25
Port: <none>
Host Port: <none>
Command:
sh
Args:
-c
sh -x /config-unicorn/configure ; sh -x /config-workhorse/configure ; mkdir -p -m 3770 /tmp/gitlab
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 16 Apr 2020 12:09:10 -0700
Finished: Thu, 16 Apr 2020 12:09:10 -0700
Ready: True
Restart Count: 0
Requests:
cpu: 50m
Environment: <none>
Mounts:
/config-unicorn from unicorn-config (ro)
/config-workhorse from workhorse-config (ro)
/init-config from init-unicorn-secrets (ro)
/init-secrets from unicorn-secrets (rw)
/init-secrets-workhorse from workhorse-secrets (rw)
/tmp from shared-tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nh99g (ro)
dependencies:
Container ID: docker://5f1e095bbacc1b8ac5dd7389bbe8506868682363a534d6c6eb5ae1f7494f8430
Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v12.9.3
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee@sha256:68523acb0b17f5e72c1981b2b37f4045448447210db378af5cba7e25400e9768
Port: <none>
Host Port: <none>
Args:
/scripts/wait-for-deps
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 16 Apr 2020 12:59:43 -0700
Finished: Thu, 16 Apr 2020 13:00:23 -0700
Ready: False
Restart Count: 13
Requests:
cpu: 50m
Environment:
GITALY_FEATURE_DEFAULT_ON: 1
CONFIG_TEMPLATE_DIRECTORY: /var/opt/gitlab/templates
CONFIG_DIRECTORY: /srv/gitlab/config
WORKHORSE_ARCHIVE_CACHE_DISABLED: 1
ENABLE_BOOTSNAP: 1
Mounts:
/etc/gitlab from unicorn-secrets (ro)
/srv/gitlab/config/secrets.yml from unicorn-secrets (ro,path="rails-secrets/secrets.yml")
/var/opt/gitlab/templates from unicorn-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nh99g (ro)
Containers:
unicorn:
Container ID:
Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v12.9.3
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 300m
memory: 1500M
Liveness: http-get http://:8080/-/liveness delay=20s timeout=30s period=60s #success=1 #failure=3
Readiness: http-get http://:8080/-/readiness delay=0s timeout=2s period=10s #success=1 #failure=3
Environment:
GITLAB_WEBSERVER: unicorn
TMPDIR: /tmp/gitlab
GITALY_FEATURE_DEFAULT_ON: 1
CONFIG_TEMPLATE_DIRECTORY: /var/opt/gitlab/templates
CONFIG_DIRECTORY: /srv/gitlab/config
prometheus_multiproc_dir: /metrics
ENABLE_BOOTSNAP: 1
Mounts:
/etc/gitlab from unicorn-secrets (ro)
/etc/ssl/certs/ from etc-ssl-certs (ro)
/metrics from unicorn-metrics (rw)
/srv/gitlab/INSTALLATION_TYPE from unicorn-config (rw,path="installation_type")
/srv/gitlab/config/initializers/smtp_settings.rb from unicorn-config (rw,path="smtp_settings.rb")
/srv/gitlab/config/secrets.yml from unicorn-secrets (rw,path="rails-secrets/secrets.yml")
/srv/gitlab/public/uploads/tmp from shared-upload-directory (rw)
/srv/gitlab/shared from shared-artifact-directory (rw)
/tmp from shared-tmp (rw)
/var/opt/gitlab/templates from unicorn-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nh99g (ro)
gitlab-workhorse:
Container ID:
Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ee:v12.9.3
Image ID:
Port: 8181/TCP
Host Port: 0/TCP
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 100M
Liveness: exec [/scripts/healthcheck] delay=20s timeout=30s period=60s #success=1 #failure=3
Readiness: exec [/scripts/healthcheck] delay=0s timeout=2s period=10s #success=1 #failure=3
Environment:
TMPDIR: /tmp/gitlab
GITLAB_WORKHORSE_EXTRA_ARGS:
GITLAB_WORKHORSE_LISTEN_PORT: 8181
CONFIG_TEMPLATE_DIRECTORY: /var/opt/gitlab/templates
CONFIG_DIRECTORY: /srv/gitlab/config
Mounts:
/etc/gitlab from workhorse-secrets (ro)
/etc/ssl/certs/ from etc-ssl-certs (ro)
/srv/gitlab/public/uploads/tmp from shared-upload-directory (rw)
/srv/gitlab/shared from shared-artifact-directory (rw)
/tmp from shared-tmp (rw)
/var/opt/gitlab/templates from workhorse-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-eeee (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
shared-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
unicorn-metrics:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
unicorn-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-ee-unicorn
Optional: false
workhorse-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-ee-workhorse-config
Optional: false
init-unicorn-secrets:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: gitlab-ee-rails-secret
SecretOptionalName: <nil>
SecretName: gitlab-ee-gitlab-shell-secret
SecretOptionalName: <nil>
SecretName: gitlab-ee-gitaly-secret
SecretOptionalName: <nil>
SecretName: gitlab-ee-redis-secret
SecretOptionalName: <nil>
SecretName: gitlab-ee-postgresql-password
SecretOptionalName: <nil>
SecretName: gitlab-ee-registry-secret
SecretOptionalName: <nil>
SecretName: gitlab-ee-gitlab-workhorse-secret
SecretOptionalName: <nil>
SecretName: gitlab-ee-minio-secret
SecretOptionalName: <nil>
SecretName: oauth2-generic
SecretOptionalName: <nil>
unicorn-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
workhorse-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
shared-upload-directory:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
shared-artifact-directory:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
etc-ssl-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
custom-ca-certificates:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: gitlab-ee-wildcard-tls-ca
SecretOptionalName: <nil>
default-token-nh99g:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-eeee
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 52m default-scheduler Successfully assigned gitlab-ee/gitlab-ee-unicorn-zzzzz to ip.xxx.xxx.xxx
Normal Pulled 52m kubelet, ip.xxx.xxx.xxx Container image "registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20171114-r3" already present on machine
Normal Created 52m kubelet, ip.xxx.xxx.xxx Created container certificates
Normal Started 52m kubelet, ip.xxx.xxx.xxx Started container certificates
Normal Pulling 52m kubelet, ip.xxx.xxx.xxx Pulling image "busybox:latest"
Normal Pulled 52m kubelet, ip.xxx.xxx.xxx Successfully pulled image "busybox:latest"
Normal Created 52m kubelet, ip.xxx.xxx.xxx Created container configure
Normal Started 52m kubelet, ip.xxx.xxx.xxx Started container configure
Normal Created 49m (x4 over 52m) kubelet, ip.xxx.xxx.xxx Created container dependencies
Normal Started 49m (x4 over 52m) kubelet, ip.xxx.xxx.xxx Started container dependencies
Normal Pulled 42m (x7 over 52m) kubelet, ip.xxx.xxx.xxx Container image "registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:v12.9.3" already present on machine
Warning BackOff 2m32s (x192 over 51m) kubelet, ip.xxx.xxx.xxx Back-off restarting failed containerEdited by Jason Plum