Investigate sporadic Redis::CommandError NOAUTH Authentication required on gprd-cny during redis gem upgrade
Follow-up to production#17553 (comment 1765452846)
Summary of known facts:
-
redis
gem upgraded fromv4.8.0
tov5.0.8
(why not smaller version bumps? explained in gitlab-org/gitlab!141213 (comment 1731748114)) - Affected Redis instances: all. Sentinel-based deployments are on Redis 6.2 while Cluster-based deployments are on Redis 7.0
- Raised error was user-facing as web service dashboard showed corresponding increase in 5xx
Sentry links
- ~4k events of
Redis::CommandError NOAUTH Authentication required. / RedisClient::CommandError NOAUTH Authentication required.
: https://new-sentry.gitlab.net/organizations/gitlab/issues/655796/?project=3&query=release:f1f30803a7b - ~100 events of
Redis::CannotConnectError WRONGPASS invalid username-password pair or user is disabled. RedisClient::AuthenticationError WRONGPASS invalid username-password pair or user is disabled.
: https://new-sentry.gitlab.net/organizations/gitlab/issues/655744/?project=3&query=release%3Af1f30803a7b&referrer=release-issue-stream
Local reproduction of error
The first step is to ascertain the conditions required to get the errors reflected above.
Using a local Redis server with requirepass
, NOAUTH Authentication required
is produced when password is set to nil
while WRONGPASS
is produced if the password is an incorrect string.
[10] pry(main)> Redis.new(Gitlab::Redis::SharedState.params.merge(password: nil)).ping
Redis::CommandError: NOAUTH Authentication required.
from /Users/sylvesterchin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/redis-client-0.19.0/lib/redis_client/connection_mixin.rb:35:in `call'
Caused by RedisClient::CommandError: NOAUTH Authentication required.
from /Users/sylvesterchin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/redis-client-0.19.0/lib/redis_client/connection_mixin.rb:35:in `call'
[11] pry(main)> Redis.new(Gitlab::Redis::SharedState.params.merge(password: '')).ping
Redis::CannotConnectError: WRONGPASS invalid username-password pair or user is disabled.
from /Users/sylvesterchin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/redis-client-0.19.0/lib/redis_client/connection_mixin.rb:61:in `call_pipelined'
Caused by RedisClient::AuthenticationError: WRONGPASS invalid username-password pair or user is disabled.
from /Users/sylvesterchin/.local/share/mise/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/redis-client-0.19.0/lib/redis_client/connection_mixin.rb:61:in `call_pipelined'
The second part is to reproduce the errors in gprd-cny
without intentionally setting a wrong password, but purely by existing application-side behaviours.
What we tried:
- Forking. There were seemingly related open issues (https://github.com/redis/redis-rb/issues/909) mentioning errors while forking however the issue is quite old and relevant to the v4.3 of the redis gem. Forking the existing client works fine.
[44] pry(main)> fork { puts rc.ping }
=> 4417
[45] pry(main)> PONG
Shutting down background worker
Killing session flusher
- Closing and re-establishing the connection works (as it should). Verified using
redis-cli client list
on the local test server.
[45] pry(main)> rc.close
=> nil
[46] pry(main)> rc.ping
=> "PONG"
Edited by Sylvester Chin