Stuck in SAML Loop after upgrading from chart 4.5.3 to 4.7.5
Summary
Hi all, I'm currently trying to upgrade the gitlab installation from chart version 4.5.3 to chart version 4.7.5.
At first, I found the warning about the deprecated force-namespace-isolation, so I remove it from the values file.
After rerunning the upgrade for the second time, it was success, but when I tried to access the UI, it returns ERR_TO_MANY_REDIRECTS. For this issue, I fixed by adding nginx.ingress.kubernetes.io/ssl-redirect: false at the ingress annotation.
Then finally, when trying to access the UI again, after changing the configurations, it redirect me to the SSO login page (I'm using keycloak as SSO provider).
After successfully authenticate the user via SSO, I was stuck in the state which gitlab redirect me back to the SSO provider, then the provider redirect to the gitlab in an endless loop.
I've already tried the following:
-
Set global.hosts.https to false. This allows me to not stuck in the loop after successfully authenticate. But somehow the provider redirect me to http:// instead of https://. In the older chart version, the provider always redirect me to https:// after the authentication.
-
Try upgrading from chart 4.5.3 to latest chart version 4.6.x using the same configurations. This makes me able to login to gitlab via SSO provider and not stuck in the SAML loop. It also redirect me to https://, which is the correct behavior.
I was unsure if this issue happens because the nginx version was upgraded in chart 4.7.x or not.
Please help to advice for this issue, as we need to upgrade our gitlab to the latest version.
Steps to reproduce
Run helm upgrade command with the provided values file, using chart version 4.7.5 (or any version after 4.7.0)
Configuration used
### values.yaml ###
global:
operator:
enabled: false
rollout:
autoPause: true
edition: ce
application:
create: false
links: []
allowClusterRoles: true
hosts:
domain: sample.domain.com
https: true
hostSuffix: staging
ingress:
enabled: true
configureCertmanager: true
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: false
gitlab:
license: {}
psql:
port: 5432
database: gitlab-sample-db-name
username: dbuser
password:
secret: gitlab-postgres
key: psql-password
redis:
password:
enabled: true
gitaly:
authToken: {}
internal:
names: ['default']
external: []
tls:
enabled: false
minio:
enabled: false
credentials: {}
grafana:
enabled: false
appConfig:
enableUsagePing: true
enableImpersonation: true
defaultCanCreateGroup: true
usernameChangingEnabled: false
issueClosingPattern:
defaultTheme:
defaultProjectsFeatures:
issues: true
mergeRequests: true
wiki: true
snippets: true
builds: true
webhookTimeout:
cron_jobs: {}
object_store:
enabled: true
proxy_download: true
storage_options:
server_side_encryption: aws:kms
server_side_encryption_kms_key_id: my-kms-key
connection:
secret: gitlab-rails-storage
key: connection
gravatar:
plainUrl:
sslUrl:
extra:
googleAnalyticsId:
piwikUrl:
piwikSiteId:
lfs:
bucket: my-lfs-s3-bucket
connection: {}
artifacts:
bucket: my-artifacts-s3-bucket
connection: {}
uploads:
bucket: my-uploads-s3-bucket
connection: {}
packages:
bucket: my-packages-s3-bucket
connection: {}
backups:
bucket: my-backup-s3-bucket
tmpBucket: my-tmp-s3-bucket
incomingEmail:
enabled: false
address: ""
host: "imap.gmail.com"
port: 993
ssl: true
startTls: false
user: ""
password:
secret: ""
key: password
mailbox: inbox
idleTimeout: 60
ldap:
preventSignin: false
servers: {}
omniauth:
enabled: true
autoSignInWithProvider: saml
syncProfileFromProvider: ['saml']
allowSingleSignOn: ['saml']
blockAutoCreatedUsers: false
autoLinkLdapUser: false
autoLinkSamlUser: true
providers:
- secret: gitlab-saml
key: provider
geo:
enabled: false
role: primary
nodeName: # defaults to `gitlab.gitlab.host`
psql:
password: {}
shell:
install: true
enabled: true
authToken: {}
hostKeys: {}
port: 15022
railsSecrets: {}
registry:
enabled: true
certificate: {}
httpSecret: {}
bucket: my-registry-s3-bucket
storage:
secret: registry-storage
key: config
runner:
registrationToken: {}
smtp:
enabled: false
address: smtp.mailgun.org
port: 2525
user_name: ""
password:
secret: ""
key: password
authentication: "plain"
starttls_auto: false
openssl_verify_mode: "peer"
email:
from: ''
display_name: GitLab
reply_to: ''
subject_suffix: ''
smime:
enabled: false
secretName: ""
keyName: "tls.key"
certName: "tls.crt"
time_zone: "Asia/Bangkok"
service:
annotations: {}
antiAffinity: soft
workhorse: {}
certificates:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates
tag: 20171114-r3
customCAs: []
kubectl:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/kubectl
tag: v1.12.10
pullSecrets: []
upgradeCheck:
enabled: true
image:
repository: busybox
tag: latest
tolerations: []
resources:
requests:
cpu: 50m
certmanager-issuer:
email: my-email@example.com
certmanager:
install: true
rbac:
create: true
nginx-ingress:
enabled: true
tcpExternalConfig: "true"
controller:
config:
hsts-include-subdomains: "false"
server-name-hash-bucket-size: "256"
enable-vts-status: "true"
use-http2: "false"
ssl-ciphers: "my-ssl-cipher"
ssl-protocols: "TLSv1.3 TLSv1.2"
server-tokens: "false"
service:
targetPorts:
http: http
https: http
enableHttp: "false"
externalTrafficPolicy: "Local"
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: my-acm-cert
service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-2017-01
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
service.beta.kubernetes.io/aws-load-balancer-access-log-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name: "my-access-log-s3-bucket"
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix: "log-prefix"
resources:
requests:
cpu: 100m
memory: 100Mi
publishService:
enabled: true
replicaCount: 3
minAvailable: 2
scope:
enabled: true
stats:
enabled: true
metrics:
enabled: true
service:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "10254"
defaultBackend:
minAvailable: 1
replicaCount: 2
resources:
requests:
cpu: 5m
memory: 5Mi
rbac:
create: true
serviceAccount:
create: true
prometheus:
install: true
rbac:
create: true
alertmanager:
enabled: false
alertmanagerFiles:
alertmanager.yml: {}
kubeStateMetrics:
enabled: false
nodeExporter:
enabled: false
pushgateway:
enabled: false
server:
statefulSet:
enabled: true
persistentVolume:
storageClass: my-ebs-storage-class
redis:
install: true
persistence:
storageClass: my-ebs-storage-class
redis-ha:
install: false
nameOverride: redis
postgresql:
install: false
shared-secrets:
enabled: true
rbac:
create: true
gitlab-runner:
install: false
gitlab:
gitlab-shell:
install: true
enabled: true
gitaly:
install: true
persistence:
storageClass: my-ebs-storage-class
migrations:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce
sidekiq:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce
webservice:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ce
workhorse:
image: registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce
task-runner:
image:
repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce
persistent:
storageClass: my-ebs-storage-class
backups:
objectStorage:
config:
secret: s3cmd-config
key: config
### provider.yaml ###
name: 'saml'
label: 'Keycloak SAML'
groups_attribute: 'roles'
args:
assertion_consumer_service_url: 'https://gitlab-staging.sample.domain.com/users/auth/saml/callback'
idp_cert_fingerprint: 'XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX'
idp_sso_target_url: 'https://keycloak.sample.domain.com/auth/realms/master/protocol/saml/clients/gitlab'
issuer: 'gitlab'
attribute_statements:
first_name: ['first_name']
last_name: ['last_name']
name: ['name']
username: ['name']
email: ['email']
name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent'
Current behavior
After authenticating the user via SSO Provider, the user was stuck in an endless loop where gitlab and the sso provider keeps redirecting to each other.
Expected behavior
The user should be able to login via the SSO provider, the provider should redirect the user to the gitlab UI without stucking in an authentication loop.
Versions
- Chart: 4.7.5 (Or any version above 4.7.0)
- Platform:
- EKS
- Kubernetes: (
kubectl version
)- Server: 1.18.9
- Helm: (
helm version
)- Client: 2.16.9