Can't verify CSRF token authenticity - 422 AWS EKS

Summary

Hello all, I have deployed Gitlab CE on our EKS cluster in AWS. Using helm values below, and I believe it deployed ok, at least I do not see anything stuck. It was deployed with ArgoCD (via Terraform pushed to ArgoCD) and I can see all "green" there. However, I’m facing an issue when I try to log in. The traffic currently flow like this:

DNS(gtilab.domain.com, managed on Azure) --> AWS ALB CNAME(SSL terminated here, with wild certificate) --> Traefik Ingress --(HTTP traffic)--> gitlab-webservice-default(8181)

However I get the 422 redirect if I try to register new user, or log in with non existing user, or with admin/pass and in log I can see Can't verify CSRF token authenticity Since start I did not wanted to use the build in nginx since we do not use it, and no SSL since the termination is happening on ALB. I spend a lot of time googling the issue, but nothing seems to help, I'm at the end of my rope.

I have also noticed that when I get the 422 page, the certificate notice pops up that not all content is secured.... which is strange, everything should be using https://

Steps to reproduce

EKS deployment with traefik and ALB and bellow config.

Configuration used

global:
  serviceAccount:
    enabled: true
    create: false
    name: aws-access
  platform:
    eksRoleArn: ${aws_iam_role}
  shell:
    authToken:
      secret: ${shell_secret}
      key: password
  ## https://docs.gitlab.com/charts/installation/deployment#deploy-the-community-edition
  edition: ce
  gitlab:
    ## Enterprise license for this GitLab installation
    ## Secret created according to https://docs.gitlab.com/charts/installation/secrets#initial-enterprise-license
    ## If allowing shared-secrets generation, this is OPTIONAL.
    license: {}
      # secret: RELEASE-gitlab-license
      # key: license
  tls:
    enabled: false
  hosts:
    domain: ${gitlab_dns_name}
    https: true
    externalIP:
    gitlab:
      name: gitlab.${gitlab_dns_name}
      https: true
    registry:
      name: registry.${gitlab_dns_name}
      https: true
    pages:
      name: pages.${gitlab_dns_name}
      https: true
    ssh: ~
  ingress:
    enabled: false
    tls:
      enabled: false
    configureCertmanager: false
  psql:
    password:
      useSecret: true
      secret: ${psql_password}
      key: psql-password
    host: ${psql_host}
    port: ${psql_port}
    username: ${psql_user_name}
    database: ${psql_db_name}
  redis:
    password:
      enabled: false
    host: ${redis_host}
  minio:
    enabled: false
  appConfig:
    enableUsagePing: false
    backups:
      bucket: ${s3_backup}
    lfs:
      enabled: true
      bucket: ${s3_lfs}
      connection:
        secret: ${s3_connection_secret}
        key: connection
    artifacts:
      enabled: true
      bucket: ${s3_artifacts}
      connection:
        secret: ${s3_connection_secret}
        key: connection
    uploads:
      enabled: true
      bucket: ${s3_upload}
      connection:
        secret: ${s3_connection_secret}
        key: connection
    packages:
      enabled: true
      bucket: ${s3_packages}
      connection:
        secret: ${s3_connection_secret}
        key: connection
    terraformState:
      enabled: true
      bucket: ${s3_terrafrom_state}
      connection:
        secret: ${s3_connection_secret}
        key: connection
nginx-ingress:
  enabled: false
certmanager:
  install: false
  installCRDs: false
  rbac:
    create: false
prometheus:
  install: false
redis:
  install: false
upgradeCheck:
  enabled: true
gitlab:
  gitaly:
    persistence:
      enabled: true
      storageClass: ebs-sc
      size: ${gitaly_storage_size}
      accessMode: ReadWriteOnce
  toolbox:
    backups:
      objectStorage:
        backend: s3
        config:
          secret: ${s3_connection_secret}
          key: connection
    annotations:
      eks.amazonaws.com/role-arn: ${aws_iam_role}
  webservice:
    annotations:
      eks.amazonaws.com/role-arn: ${aws_iam_role}
  sidekiq:
    annotations:
      eks.amazonaws.com/role-arn: ${aws_iam_role}
  migrations:
    psql:
      password:
        secret: ${psql_password}
        key: psql-password
      host: ${psql_host}
      port: ${psql_port}
postgresql:
  install: false
gitlab-runner:
  install: true
  rbac:
    create: true
  runners:
    locked: false
registry:
  enabled: true
  annotations:
    eks.amazonaws.com/role-arn: ${aws_iam_role}
  storage:
    secret: ${s3_reistry_connection_secret}
    key: config

Ingres (ALB pointing to traefik)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: traefik
  name: traefik-ingress
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/certificate-arn: arn.<redacted>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internal 
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-port: '80'
    alb.ingress.kubernetes.io/healthcheck-path: /ping
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
spec:
  ingressClassName: alb
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: ssl-redirect
                port:
                  name: use-annotation          
          - path: /
            pathType: Prefix
            backend:
              service:
                name: traefik
                port:
                  number: 443

Traefik IngressRoute for gitlab.

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: gitlab-webservice
  labels:
    app: traefik
  namespace: gitlab  
spec:
  entryPoints:
  - websecure  
  routes:
  - kind: Rule
    match: Host(`gitlab.<redacted>.com`) && PathPrefix(`/admin/sidekiq`)
    services:
    - kind: Service
      name: gitlab-webservice-default
      port: 8080
  - kind: Rule
    match: Host(`gitlab.<redacted>.com`) && PathPrefix(`/`)
    services:
    - kind: Service
      name: gitlab-webservice-default
      port: 8181

Current behavior

I get log in page but when I log in I get 422 error page, including note from the browser that not all connection is secured meaning somehting is trying to use http.

In pod log I noticed this: Can't verify CSRF token authenticity

Expected behavior

Normal log in.

Versions

• Chart: 6.4.2
• Platform:
  • Cloud: EKS
• Kubernetes: (kubectl version)
  • Client Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.7-eks-4721010", GitCommit:"b77d9473a02fbfa834afa67d677fd12d690b195f", GitTreeState:"clean", BuildDate:"2022-06-27T22:22:16Z", GoVersion:"go1.17.10", Compiler:"gc", Platform:"linux/amd64"}
  • Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.10-eks-15b7512", GitCommit:"cd6399691d9b1fed9ec20c9c5e82f5993c3f42cb", GitTreeState:"clean", BuildDate:"2022-08-31T19:17:01Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

Relevant logs

*** /var/log/gitlab/production.log ***
Started POST "/users/sign_in" for 172.21.2.244 at 2022-10-10 02:42:40 +0000
Processing by SessionsController#create as HTML
  Parameters: {"authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"vstryce", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Can't verify CSRF token authenticity.
Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 448)

ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):

lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

*** /var/log/gitlab/production_json.log ***
{"method":"POST","path":"/users/sign_in","format":"html","controller":"SessionsController","action":"create","status":422,"time":"2022-10-10T02:42:40.609Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"user","value":{"login":"vstryce","password":"[FILTERED]","remember_me":"0"}}],"remote_ip":"172.21.2.244","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:105.0) Gecko/20100101 Firefox/105.0","queue_duration_s":0.07713,"request_urgency":"low","target_duration_s":5,"redis_calls":3,"redis_duration_s":0.001737,"redis_read_bytes":608,"redis_write_bytes":198,"redis_cache_calls":3,"redis_cache_duration_s":0.001737,"redis_cache_read_bytes":608,"redis_cache_write_bytes":198,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_main_count":1,"db_main_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_main_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_main_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"db_main_duration_s":0.006,"db_main_replica_duration_s":0.0,"cpu_s":0.020398,"mem_objects":8894,"mem_bytes":947059,"mem_mallocs":2572,"mem_total_bytes":1302819,"pid":34,"worker_id":"puma_0","rate_limiting_gates":[],"correlation_id":"01GEZWBRRF96G38FHP8Z2YD8JY","exception.class":"ActionController::InvalidAuthenticityToken","exception.message":"ActionController::InvalidAuthenticityToken","exception.backtrace":["lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:26:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'","lib/gitlab/middleware/release_env.rb:13:in `call'"],"db_duration_s":0.0,"view_duration_s":0.0,"duration_s":0.00115}