Runner won't register with minio set to false OpenSSL::Cipher::CipherError

Summary

Runner won't register with minio set to false (external object-store) with a Can't verify CSRF token authenticity. / OpenSSL::Cipher::CipherError () error.

GitLab does work fine (with a working HTTPS via a cluster-wide certmanager/ingress-nginx setup). Having no issues when using the bundled minio.

A new install does not have any issues so I guess it must be something after restoring the backup.

Steps to reproduce

Deploy helm chart:

CHART_VERSION:="4.11.1"
EXTERNAL_INGRESS_NGING_CHART_VERSION:="3.29.0"
EXTERNAL_CERT_MANAGER_CHART_VERSION:="1.3.1"

Configuration used

Merge of those two config is used:

global:
  hosts:
    domain: foo.io
    # hostSuffix: staging
    externalIP: 1.2.3.4
  smtp:
    enabled: true
    address: in-v3.mailjet.com
    port: 587
    user_name: xxx
    password:
      secret: gitlab-smtp-password # https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/installation/secrets.md#smtp-password
      key: password
  email:
    from: gitlab@foo.io
    display_name: GitLab Foo
    reply_to: noreply@foo.io
    subject_suffix: 'Foo'
  minio:
    enabled: false
  registry:
    bucket: gitlab-registry-storage-foo-io
  appConfig:
    object_store:
      enabled: true
      proxy_download: true
      storage_options:
        {}
        # server_side_encryption:
        # server_side_encryption_kms_key_id
      connection:
        secret: objectstore-rails
        key: rails.s3.yaml
registry:
  storage:
    secret: objectstore-registry
    key: registry.s3.yaml
gitlab-runner:
  runners:
    cache:
      s3ServerAddress: s3.gra.cloud.ovh.net
      s3BucketName: gitlab-runner-cache
      s3BucketLocation: gra
      secretName: objectstore-runner

certmanager:
  install: false

nginx-ingress:
  enabled: false

global:
  edition: ce
  ingress:
    class: nginx
    configureCertmanager: false
    annotations:
      kubernetes.io/tls-acme: 'true'
      cert-manager.io/cluster-issuer: letsencrypt
  minio:
    enabled: true

gitlab:
  webservice:
    ingress:
      tls:
        secretName: 'gitlab-webservice-tls'
  task-runner:
    backups:
      cron:
        enabled: true
        schedule: '5 5 * * 1'
      objectStorage:
        config:
          secret: gitlab-backup-config
          key: backup.s3cfg

registry:
  ingress:
    tls:
      secretName: 'gitlab-registry-tls'

minio:
  persistence:
    size: 32Gi
  ingress:
    tls:
      secretName: 'gitlab-minio-tls'

gitlab-runner:
  runners:
    privileged: false

Current behavior

Runner crashes:


Registration attempt 1 of 30
Runtime platform                                    arch=amd64 os=linux pid=15 revision=7f7a4bb0 version=13.11.0
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing: 
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...                   
                                                   
ERROR: Registering runner... failed                 runner=oBcyown8 status=500 Internal Server Error
PANIC: Failed to register the runner. You may be having network problems.

GitLab logs:

{"severity":"ERROR","time":"2021-04-27T09:02:30.010Z","correlation_id":"01F4991K3C171YRW6ST2A0ZR5Q","exception.class":"OpenSSL::Cipher::CipherError","exception.message":"","exception.backtrace":["lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'","app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:22:in `decrypt_token'","app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'","app/models/concerns/token_authenticatable_strategies/base.rb:38:in `ensure_token!'","app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field'","app/models/application_setting_implementation.rb:353:in `runners_registration_token'","lib/gitlab/current_settings.rb:28:in `method_missing'","lib/api/helpers/runner.rb:14:in `runner_registration_token_valid?'","lib/api/ci/runner.rb:34:in `block (2 levels) in \u003cclass:Runner\u003e'","lib/api/api_guard.rb:213:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/transaction.rb:56:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:172:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:21:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:76:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":"continuous_integration","tags.correlation_id":"01F4991K3C171YRW6ST2A0ZR5Q"}

Started POST "/api/v4/runners" for 10.244.0.87 at 2021-04-27 09:02:29 +0000 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `block (2 levels) in generate_api_method' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice] Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:182:in `instrument' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:58:in `block in generate_api_method' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice] Can't verify CSRF token authenticity. 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice] Completed 422 Unprocessable Entity in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms | Allocations: 205) 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:341:in `execute' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]  
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:267:in `block in run' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:180:in `block in instrument' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice] OpenSSL::Cipher::CipherError (): 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:180:in `instrument' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:247:in `run' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:322:in `block in build_stack' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:22:in `decrypt_token' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/app/models/concerns/token_authenticatable_strategies/base.rb:38:in `ensure_token!' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/app/models/concerns/token_authenticatable.rb:48:in `block in add_authentication_token_field' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/app/models/application_setting_implementation.rb:353:in `runners_registration_token' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/lib/gitlab/current_settings.rb:28:in `method_missing' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/lib/api/api_guard.rb:213:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/lib/api/helpers/runner.rb:14:in `runner_registration_token_valid?' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/lib/api/ci/runner.rb:34:in `block (2 levels) in <class:Runner>' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `call' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `block (2 levels) in generate_api_method' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/resource.rb:20:in `_call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:182:in `instrument' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:58:in `block in generate_api_method' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/abstract/handler.rb:17:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:341:in `execute' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:39:in `block in call!' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:267:in `block in run' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:38:in `catch' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:38:in `call!' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:180:in `block in instrument' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:180:in `instrument' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:247:in `run' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `catch' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `call!' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:322:in `block in build_stack' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call' 
[gitlab-webservice-default-6595c4bd5f-kbcgt webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call' 
[gitlab-webservice-default-6595c4bd5f-7f5lk webservice]   /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call' 
[...]

Expected behavior

Working registration

Versions

Chart: v4.11.1
Platform:
- Self-hosted: Kubeadm
Kubernetes: (kubectl version)
- Client: v1.21.0
- Server: v1.21.0
Helm: (helm version)
- Client: v3.5.4

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)

Edited Apr 27, 2021 by Olivier Louvignes