Skip to content

Unable to connect to Redis 6 as non-default user

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

Redis 6 introduced ACLs which give administrators more control over security and permissions in Redis (https://redis.io/topics/acl). It is currently not possible to configure a username for the Redis connection in the GitLab charts or in the omnibus configuration.

Even with some modifications to the charts that inject a username into the Redis connection string in resque.yml and cable.yml, I'm still seeing Redis authentication failures in sidekiq.

Steps to reproduce

Create a non-default user in a Redis 6 instance. Attempt to configure GitLab to connect to Redis as that user.

What is the current bug behavior?

Using this change to the charts, I updated my redis connection string to include a username, so the connection string is of the form: rediss://{user}:{password}@{host}:{port}. With this change in place, webservice starts correctly, and the dependencies container of sidekiq reports success connecting to Redis. However, the sidekiq pods still crash with an error indicating the credentials are incorrect.

What is the expected correct behavior?

The webservice and sidekiq pods should function normally, using the username and password in the connection string to connect to the Redis instance.

Relevant logs and/or screenshots

sidekiq pod logs - dependencies container

Checking: resque.yml, cable.yml
+ SUCCESS connecting to 'rediss://{HOST_REDACTED}:{PORT_REDACTED}' from cable.yml, through {HOST_REDACTED}
+ SUCCESS connecting to 'rediss://{HOST_REDACTED}:{PORT_REDACTED}' from resque.yml, through {HOST_REDACTED}

sidekiq pod logs - sidekiq container

2022-03-03T16:55:56.538Z pid=21 tid=9it INFO: Booting Sidekiq 6.3.1 with redis options {:scheme=>"rediss", :host=>"HOST_REDACTED", :port=>PORT_REDACTED, :password=>"REDACTED", :id=>nil, :instrumentation_class=>Gitlab::Instrumentation::Redis::Queues, :namespace=>"resque:gitlab"}
2022-03-03T16:55:58.970Z pid=21 tid=9it INFO: GitLab reliable fetch activated!
2022-03-03T16:55:59.040Z pid=21 tid=9it WARN: Redis::CommandError: WRONGPASS invalid username-password pair
2022-03-03T16:55:59.041Z pid=21 tid=9it WARN: /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:149:in `call'
/srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:21:in `call'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:126:in `block in connect'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:330:in `with_reconnect'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:116:in `connect'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:403:in `ensure_connected'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:255:in `block in process'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:342:in `logging'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:254:in `process'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis/client.rb:148:in `call'
/srv/gitlab/lib/gitlab/instrumentation/redis_interceptor.rb:21:in `call'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis.rb:1488:in `block in smembers'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis.rb:70:in `block in synchronize'
/usr/lib64/ruby/2.7.0/monitor.rb:202:in `synchronize'
/usr/lib64/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis.rb:70:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-4.4.0/lib/redis.rb:1487:in `smembers'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-namespace-1.8.1/lib/redis/namespace.rb:476:in `call_with_namespace'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/redis-namespace-1.8.1/lib/redis/namespace.rb:352:in `block (2 levels) in <class:Namespace>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:205:in `block in all'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq.rb:99:in `block in redis'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq.rb:96:in `redis'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:204:in `all'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:520:in `destroy_removed_jobs'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-cron-1.2.0/lib/sidekiq/cron/job.rb:163:in `load_from_hash!'
/srv/gitlab/config/initializers/sidekiq.rb:88:in `block in <main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq.rb:77:in `configure_server'
/srv/gitlab/config/initializers/sidekiq.rb:29:in `<main>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in `load'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/bootsnap-1.9.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:60:in `load'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/engine.rb:681:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/notifications.rb:205:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/engine.rb:680:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/engine.rb:634:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/engine.rb:633:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/engine.rb:633:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:32:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:32:in `run'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:61:in `block in run_initializers'
/usr/lib64/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'
/usr/lib64/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/lib64/ruby/2.7.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/usr/lib64/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/lib64/ruby/2.7.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:50:in `each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:50:in `tsort_each_child'
/usr/lib64/ruby/2.7.0/tsort.rb:415:in `call'
/usr/lib64/ruby/2.7.0/tsort.rb:415:in `each_strongly_connected_component_from'
/usr/lib64/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/lib64/ruby/2.7.0/tsort.rb:347:in `each'
/usr/lib64/ruby/2.7.0/tsort.rb:347:in `call'
/usr/lib64/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'
/usr/lib64/ruby/2.7.0/tsort.rb:226:in `tsort_each'
/usr/lib64/ruby/2.7.0/tsort.rb:205:in `tsort_each'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/initializable.rb:60:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.4/lib/rails/application.rb:391:in `initialize!'
/srv/gitlab/config/environment.rb:7:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/dependencies.rb:332:in `require'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/dependencies.rb:332:in `block in require'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/dependencies.rb:299:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.4/lib/active_support/dependencies.rb:332:in `require'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/cli.rb:273:in `boot_application'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/lib/sidekiq/cli.rb:37:in `run'
/srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sidekiq-6.3.1/bin/sidekiq:31:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.7.0/bin/sidekiq:23:in `load'
/srv/gitlab/vendor/bundle/ruby/2.7.0/bin/sidekiq:23:in `<top (required)>'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/cli/exec.rb:63:in `load'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/cli/exec.rb:63:in `kernel_load'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/cli/exec.rb:28:in `run'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/cli.rb:474:in `exec'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/cli.rb:30:in `dispatch'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/cli.rb:24:in `start'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/exe/bundle:49:in `block in <top (required)>'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/lib/bundler/friendly_errors.rb:130:in `with_friendly_errors'
/usr/lib64/ruby/gems/2.7.0/gems/bundler-2.2.19/exe/bundle:37:in `<top (required)>'
/usr/bin/bundle:23:in `load'
/usr/bin/bundle:23:in `<main>'
2022-03-03T16:55:59.041Z pid=21 tid=474l ERROR: Heartbeat thread error: WRONGPASS invalid username-password pair
{"severity":"INFO","time":"2022-03-03T16:55:59.424Z","message":"A worker terminated, shutting down the cluster"}

Additional Notes

  • When the credentials actually are incorrect, the dependencies container reports the failure to connect to Redis. With correct creds, the dependencies container reports success, but sidekiq still crashes, complaining about incorrect creds.
  • I wrote a simple Ruby program to confirm that the redis 4.4.0 gem that is used by GitLab actually does support connecting to Redis 6 with a username and password:
gem 'redis', '=4.4.0'
require "redis"

redis = Redis.new(
  :url =>  "rediss://{REDACTED_USER}:{REDACTED_PASSWORD}@{REDACTED_HOST}:{REDACTED_PORT}",
  :ssl_params => {
    :ca_file => "/tmp/rubytest/ca.crt"
  }
)
redis.get("mykey")
  • I was also able to get a standalone sidekiq v6.3.1 instance to connect using the same Redis connection string that is failing in the GitLab sidekiq pod. I confirmed this using the example at https://github.com/mperham/sidekiq/blob/main/examples/por.rb
  • I'd be interested in contributing a fix, but so far, I'm having a little trouble figuring out why sidekiq is failing to connect even though the dependencies container reports success.
Edited by 🤖 GitLab Bot 🤖