Infamous HTTP status 422 during sign in
Summary
Deploying GitLab (Helm Chart) behind TLS-terminating load balancer isn't working as expected. If a user tries to sign in, they're redirected to /users/sign_in
which only shows "422 - The change you requested was rejected."
Our setup is fairly simple and consists of a single-node MicroK8s installation on a Google Cloud VM. The VM runs behind an Application Load Balancer which also terminates TLS. Furthermore, we're installing the MicroK8s "ingress" add-on, which centrally deploys an NGINX ingress controller.
I came across a bunch of issues related to the infamous 422 in one way or another, however none applied to our particular setup or provided a clear resolution.
Steps to reproduce
- Deploy GitLab Helm Chart (v7.11.1)
- Expose ingress behind TLS-terminating Load Balancer (e.g. Google Cloud Application Load Balancer)
Configuration used
A minimal set of values that allows reproducing the issue:
global:
hosts:
domain: my.custom.domain # Placeholder for our actual domain. All subdomains, e.g. gitlab.my.custom.domain, etc., resolve to our load balancer's public IP
https: true # From what I understand, this is required for e.g. external URLs
ingress:
configureCertmanager: false
class: "public" # MicroK8s requires class "public"
tls:
enabled: false
certmanager:
install: false
nginx-ingress:
enabled: false
Current behavior
Users can't log in.
Expected behavior
Users can log in.
Versions
- Chart: v7.11.1
- Platform:
- Cloud: Google Cloud VM
- Self-hosted: MicroK8s
- Kubernetes:
- Client Version: v1.29.4
- Server Version: v1.29.2
- Helm: v3.14.4
Relevant logs
gitlab-webservice-...
logs related to CSRF token authenticity:
...
{"component": "gitlab","subcomponent":"production","time":"2024-05-01T11:52:03Z","message":"Can't verify CSRF token authenticity."}
...
{"component": "gitlab","subcomponent":"production_json","method":"POST","path":"/users/sign_in","format":"html","controller":"SessionsController","action":"create","status":422,"time":"2024-05-01T11:52:03.209Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"user","value":{"login":"root","password":"[FILTERED]",..."exception.class":"ActionController::InvalidAuthenticityToken","exception.message":"Can't verify CSRF token authenticity.","exception.backtrace":["actionpack (7.0.8.1) lib/action_controller/metal/...
...
{"component": "gitlab","subcomponent":"production","time":"2024-05-01T11:52:03Z","message":"ActionController::InvalidAuthenticityToken (Can't verify CSRF token authenticity.):"}
...