422 Error on login to Gitlab

Hello Team, I installed Gitlab latest version for 2k architecture using GET tool on ubuntu 22.04 VMs and everything went well until I tried to login to GitLab using the UI, and it returned 422 The change you requested was rejected.

I looked into the logs and I found this:

{“method”:“POST”,“path”:“/users/sign_in","format":"html","controller":"SessionsController","action":"create","status":422,"time":"2023-09-26T20:54:31.260Z","params":[{"key":"authenticity_token","value":"[FILTERED]"},{"key":"user","value":{"login":"root","password":"[FILTERED]","remember_me":"0"}}],"remote_ip":"x.x.x.x","ua":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36","queue_duration_s":0.034994,"request_urgency":"low","target_duration_s":5,"redis_calls":3,"redis_duration_s":0.001824,"redis_read_bytes":114,"redis_write_bytes":316,"redis_cache_calls":1,"redis_cache_duration_s":0.000799,"redis_cache_read_bytes":114,"redis_cache_write_bytes":96,"redis_sessions_calls":2,"redis_sessions_duration_s":0.001025,"redis_sessions_write_bytes":220,"db_count":2,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":2,"db_main_count":2,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.009,"db_main_duration_s":0.009,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.036413,"mem_objects":13985,"mem_bytes":1602224,"mem_mallocs":4337,"mem_total_bytes":2161624,"pid":846543,"worker_id":"puma_4","rate_limiting_gates":[],"correlation_id":"01HB9MB91DEDNDB8JTN70JEHDC","exception.class":"ActionController::InvalidAuthenticityToken","exception.message":"Can't verify CSRF token authenticity.","exception.backtrace":["lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in call’”,“lib/gitlab/middleware/memory_report.rb:13:in call'","lib/gitlab/middleware/speedscope.rb:13:in call’”,“lib/gitlab/database/load_balancing/rack_middleware.rb:23:in call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in call’”,“lib/gitlab/etag_caching/middleware.rb:21:in call'","lib/gitlab/metrics/rack_middleware.rb:16:in block in call’”,“lib/gitlab/metrics/web_transaction.rb:46:in run'","lib/gitlab/metrics/rack_middleware.rb:16:in call’”,“lib/gitlab/jira/middleware.rb:19:in call'","lib/gitlab/middleware/go.rb:20:in call’”,“lib/gitlab/middleware/query_analyzer.rb:11:in block in call'","lib/gitlab/database/query_analyzer.rb:37:in within’”,“lib/gitlab/middleware/query_analyzer.rb:11:in call'","lib/gitlab/middleware/multipart.rb:173:in call’”,“lib/gitlab/middleware/read_only/controller.rb:50:in call'","lib/gitlab/middleware/read_only.rb:18:in call’”,“lib/gitlab/middleware/same_site_cookies.rb:27:in call'","lib/gitlab/middleware/basic_health_check.rb:25:in call’”,“lib/gitlab/middleware/handle_malformed_strings.rb:21:in call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in call’”,“lib/gitlab/middleware/request_context.rb:15:in call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in call’”,“config/initializers/fix_local_cache_middleware.rb:11:in call'","lib/gitlab/middleware/compressed_json.rb:44:in call’”,“lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in call’”,“lib/gitlab/metrics/requests_rack_middleware.rb:79:in call'","lib/gitlab/middleware/release_env.rb:13:in call’”],“db_duration_s”:0.00199,“view_duration_s”:0.0,“duration_s”:0.00608}

I changed nothing in the configuration of gitlab on the different nodes after the installation. gitlab is not configured with HTTPs for the moment. so the problem is with http. I tried cookies deletion and testing different browsers but none of those helped.

this is the inventory I'm using

all:
  children:
    gitaly:
      children:
        gitaly_primary:
          hosts:
            gitlab-gitaly-1:
              ansible_host: x.x.x.x
    gitlab_nfs:
      hosts:
        gitlab-nfs-1:
          ansible_host: x.x.x.x
    gitlab_rails:
      children:
        gitlab_rails_primary:
          hosts:
            gitlab-rails-1:
              ansible_host: x.x.x.x
        gitlab_rails_secondary:
          hosts:
            gitlab-rails-2:
              ansible_host: x.x.x.x
            gitlab-rails-3:
              ansible_host: x.x.x.x
    haproxy_external:
      hosts:
        gitlab-haproxy-external-1:
          ansible_host: x.x.x.x
    monitor:
      hosts:
        gitlab-monitor-1:
          ansible_host: x.x.x.x
    postgres:
      children:
        postgres_primary:
          hosts:
            gitlab-postgres-1:
              ansible_host:  x.x.x.x
    redis_persistent:
      children:
        redis_persistent_primary:
          hosts:
            gitlab-redis-persistent-1:
              ansible_host:  x.x.x.x
    ungrouped:

and this is the vars file:

all:
  vars:
    # Ansible Settings
    ansible_user: "dorra" # Typically ubuntu or ec2-user

    # Cloud Settings
    cloud_provider: "none"


    # General Settings
    external_url: "http://extbalancer"

    # Passwords / Secrets
    gitlab_root_password: 'xxxx'
    grafana_password: 'xxxx'
    postgres_password: 'xxxx'
    consul_database_password: 'xxxx'
    gitaly_token: 'xxxx'
    pgbouncer_password: 'xxxx'
    redis_password: 'xxxx'
    praefect_external_token: 'xxxx'
    praefect_internal_token: 'xxxx'
    praefect_postgres_password: 'xxxx'

Any ideas on how to resolve that?

Thanks.

Edited by Dorra Ben Letayfa