webservice chart hardcodes http
scheme in GITLAB_WORKHORSE_AUTH_BACKEND
Summary
The chart Deployment template for the gitlab-workhorse
container does not use HTTPS for the GITLAB_WORKHORSE_AUTH_BACKEND
environment variable value when the global.workhorse.tls.enabled
setting is set to true
. See https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/webservice/templates/deployment.yaml?ref_type=heads#L375-376
This leads to the dreaded "422: The change you requested was rejected" error page when attempting to log in because proxying the external request to http://localhost:8080
does not yield the necessary Set-Cookie: _gitlab_session=...
header in the response.
For example, from inside the gitlab-workhorse
container:
git@gitlab-webservice-default-5bd9f67f6d-4nbqs:/$ curl -IL http://localhost:8080/
HTTP/1.0 302 Found
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Location: http://localhost:8080/users/sign_in
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Content-Security-Policy:
NEL: {"max_age": 0}
X-Runtime: 0.060695
X-Gitlab-Meta: {"correlation_id":"32258a34-2159-4add-bdff-bbe6d22b24f8","version":"1"}
X-Request-Id: 32258a34-2159-4add-bdff-bbe6d22b24f8
Content-Length: 0
HTTP/1.0 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=utf-8
Vary: Accept
ETag: W/"f4e158dc4a76b5f62029dd2253564b90"
Cache-Control: max-age=0, private, must-revalidate
Content-Security-Policy:
NEL: {"max_age": 0}
Set-Cookie: preferred_language=en; path=/; Secure; SameSite=None
X-Runtime: 0.097153
X-Gitlab-Meta: {"correlation_id":"5c1cf8c9-f359-46d9-8879-4716e70582af","version":"1"}
X-Request-Id: 5c1cf8c9-f359-46d9-8879-4716e70582af
Content-Length: 0
Using the HTTPS endpoint does yield the required session cookie:
git@gitlab-webservice-default-5bd9f67f6d-4nbqs:/$ curl -kIL https://localhost:8081/
HTTP/1.0 302 Found
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Location: https://localhost:8081/users/sign_in
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Content-Security-Policy:
NEL: {"max_age": 0}
X-Runtime: 0.063943
X-Gitlab-Meta: {"correlation_id":"a96c6c7d-698c-4015-ae81-488a6b502a22","version":"1"}
X-Request-Id: a96c6c7d-698c-4015-ae81-488a6b502a22
Content-Length: 0
HTTP/1.0 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=utf-8
Vary: Accept
ETag: W/"1b1270d5213f2556bc4318f8b92fe8be"
Cache-Control: max-age=0, private, must-revalidate
Content-Security-Policy:
NEL: {"max_age": 0}
Set-Cookie: preferred_language=en; path=/; Secure; SameSite=None
Set-Cookie: _gitlab_session=d434e45bd7518fb211ee936f564b5b06; path=/; secure; HttpOnly; SameSite=None
X-Runtime: 0.100303
X-Gitlab-Meta: {"correlation_id":"7efaec0a-66eb-4b81-94e6-a0c5b0a6be29","version":"1"}
X-Request-Id: 7efaec0a-66eb-4b81-94e6-a0c5b0a6be29
Content-Length: 0
I was able to find a workaround by using the extraArgs
value to set -authBackend https://localhost:8081
, which luckily overrides the GITLAB_WORKHORSE_AUTH_BACKEND
environment variable value.
Steps to reproduce
GitLab webservice subchart is currently hardcoded to use http
scheme in the GITLAB_WORKHORSE_AUTH_BACKEND
environment variable value.
Configuration used
global:
# https://docs.gitlab.com/charts/charts/globals#custom-certificate-authorities
certificates:
customCAs:
- secret: gitlab-ca-certs
keys: ["ca.crt"]
# https://docs.gitlab.com/charts/charts/globals#configure-ingress-settings
ingress:
enabled: true
configureCertmanager: false
tls:
enabled: true
secretName: gitlab-webservice-tls
class: nginx
annotations:
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
# https://docs.gitlab.com/charts/charts/globals#configure-workhorse-settings
workhorse:
scheme: https
tls:
enabled: true
gitlab:
# https://docs.gitlab.com/charts/charts/gitlab/webservice
webservice:
enabled: true
http:
enabled: true # expose port 8080
tls:
enabled: true # expose port 8081
secretName: gitlab-webservice-tls
workhorse:
tls: # required when global.workhorse.tls.enabled
secretName: gitlab-webservice-tls
caSecretName: gitlab-ca-certs
# THIS IS A WORKAROUND!
extraArgs: >-
-authBackend https://localhost:8081
Current behavior
Because the workhorse container is proxying the external HTTPS request to the webservice container over HTTP, the webservice response does not set the _gitlab_session
cookie, and login (or signup, etc.) fails with the "422: The change you requested was rejected" error page.
Expected behavior
Instead of the current template hardcoding the http
scheme:
env:
- name: GITLAB_WORKHORSE_AUTH_BACKEND
value: "http://localhost:{{ $.Values.service.internalPort }}"
The template should honor the global.workhorse.tls.enabled
setting or use
.Values.workhorse.scheme
or .Values.global.workhorse.scheme
values directly:
env:
- name: GITLAB_WORKHORSE_AUTH_BACKEND
value: "{{ template "gitlab.workhorse.scheme" . }}://localhost:{{ ternary $.Values.service.tls.internalPort $.Values.service.internalPort $.Values.tls.enabled }}"
IMPORTANT (should be documented): Using the https
scheme would then require localhost
to be added as a SAN DNS name to the TLS certificate referenced by gitlab.webservice.tls.secretName
in the values file, unless workhorse is modified to safely disable certificate verification for localhost connections.
Versions
- Chart: 9.2.4 (https://gitlab.com/gitlab-org/charts/gitlab)
- Platform:
- Self-hosted: Rancher RKE2
- Kubernetes:
- Server: v1.32.7+rke2r1
- Helm: (
helm version
)- Client: v3.18.6