Skip to content

webservice chart hardcodes http scheme in GITLAB_WORKHORSE_AUTH_BACKEND

Summary

The chart Deployment template for the gitlab-workhorse container does not use HTTPS for the GITLAB_WORKHORSE_AUTH_BACKEND environment variable value when the global.workhorse.tls.enabled setting is set to true. See https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/webservice/templates/deployment.yaml?ref_type=heads#L375-376

This leads to the dreaded "422: The change you requested was rejected" error page when attempting to log in because proxying the external request to http://localhost:8080 does not yield the necessary Set-Cookie: _gitlab_session=... header in the response.

For example, from inside the gitlab-workhorse container:

git@gitlab-webservice-default-5bd9f67f6d-4nbqs:/$ curl -IL http://localhost:8080/
HTTP/1.0 302 Found
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Location: http://localhost:8080/users/sign_in
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Content-Security-Policy:
NEL: {"max_age": 0}
X-Runtime: 0.060695
X-Gitlab-Meta: {"correlation_id":"32258a34-2159-4add-bdff-bbe6d22b24f8","version":"1"}
X-Request-Id: 32258a34-2159-4add-bdff-bbe6d22b24f8
Content-Length: 0

HTTP/1.0 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=utf-8
Vary: Accept
ETag: W/"f4e158dc4a76b5f62029dd2253564b90"
Cache-Control: max-age=0, private, must-revalidate
Content-Security-Policy:
NEL: {"max_age": 0}
Set-Cookie: preferred_language=en; path=/; Secure; SameSite=None
X-Runtime: 0.097153
X-Gitlab-Meta: {"correlation_id":"5c1cf8c9-f359-46d9-8879-4716e70582af","version":"1"}
X-Request-Id: 5c1cf8c9-f359-46d9-8879-4716e70582af
Content-Length: 0

Using the HTTPS endpoint does yield the required session cookie:

git@gitlab-webservice-default-5bd9f67f6d-4nbqs:/$ curl -kIL https://localhost:8081/
HTTP/1.0 302 Found
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Location: https://localhost:8081/users/sign_in
Content-Type: text/html; charset=utf-8
Cache-Control: no-cache
Content-Security-Policy:
NEL: {"max_age": 0}
X-Runtime: 0.063943
X-Gitlab-Meta: {"correlation_id":"a96c6c7d-698c-4015-ae81-488a6b502a22","version":"1"}
X-Request-Id: a96c6c7d-698c-4015-ae81-488a6b502a22
Content-Length: 0

HTTP/1.0 200 OK
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: strict-origin-when-cross-origin
Permissions-Policy: interest-cohort=()
X-UA-Compatible: IE=edge
Content-Type: text/html; charset=utf-8
Vary: Accept
ETag: W/"1b1270d5213f2556bc4318f8b92fe8be"
Cache-Control: max-age=0, private, must-revalidate
Content-Security-Policy:
NEL: {"max_age": 0}
Set-Cookie: preferred_language=en; path=/; Secure; SameSite=None
Set-Cookie: _gitlab_session=d434e45bd7518fb211ee936f564b5b06; path=/; secure; HttpOnly; SameSite=None
X-Runtime: 0.100303
X-Gitlab-Meta: {"correlation_id":"7efaec0a-66eb-4b81-94e6-a0c5b0a6be29","version":"1"}
X-Request-Id: 7efaec0a-66eb-4b81-94e6-a0c5b0a6be29
Content-Length: 0

I was able to find a workaround by using the extraArgs value to set -authBackend https://localhost:8081, which luckily overrides the GITLAB_WORKHORSE_AUTH_BACKEND environment variable value.

Steps to reproduce

GitLab webservice subchart is currently hardcoded to use http scheme in the GITLAB_WORKHORSE_AUTH_BACKEND environment variable value.

Configuration used

global:
  # https://docs.gitlab.com/charts/charts/globals#custom-certificate-authorities
  certificates:
    customCAs:
      - secret: gitlab-ca-certs
        keys: ["ca.crt"]
  # https://docs.gitlab.com/charts/charts/globals#configure-ingress-settings
  ingress:
    enabled: true
    configureCertmanager: false
    tls:
      enabled: true
      secretName: gitlab-webservice-tls
    class: nginx
    annotations:
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  # https://docs.gitlab.com/charts/charts/globals#configure-workhorse-settings
  workhorse:
    scheme: https
    tls:
      enabled: true
gitlab:
  # https://docs.gitlab.com/charts/charts/gitlab/webservice
  webservice:
    enabled: true
    http:
      enabled: true # expose port 8080
    tls:
      enabled: true # expose port 8081
      secretName: gitlab-webservice-tls
    workhorse:
      tls: # required when global.workhorse.tls.enabled
        secretName: gitlab-webservice-tls
        caSecretName: gitlab-ca-certs
      # THIS IS A WORKAROUND!
      extraArgs: >-
        -authBackend https://localhost:8081

Current behavior

Because the workhorse container is proxying the external HTTPS request to the webservice container over HTTP, the webservice response does not set the _gitlab_session cookie, and login (or signup, etc.) fails with the "422: The change you requested was rejected" error page.

Expected behavior

Instead of the current template hardcoding the http scheme:

env:
  - name: GITLAB_WORKHORSE_AUTH_BACKEND
    value: "http://localhost:{{ $.Values.service.internalPort }}"

The template should honor the global.workhorse.tls.enabled setting or use .Values.workhorse.scheme or .Values.global.workhorse.scheme values directly:

env:
  - name: GITLAB_WORKHORSE_AUTH_BACKEND
    value: "{{ template "gitlab.workhorse.scheme" . }}://localhost:{{ ternary $.Values.service.tls.internalPort $.Values.service.internalPort $.Values.tls.enabled }}"

IMPORTANT (should be documented): Using the https scheme would then require localhost to be added as a SAN DNS name to the TLS certificate referenced by gitlab.webservice.tls.secretName in the values file, unless workhorse is modified to safely disable certificate verification for localhost connections.

Versions

Edited by Erhhung Yuan