Silent Redis::InheritedError on unicorn fork leads to nil cache reads via Redis::ConnectionError (EPIPE)
Problem
Rails.cache silences Redis errors leading to misleading nil cache reads, causing problems such as #121670 (closed). In particular we are silently hiding Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.) as well as Redis::ConnectionError (Connection lost (EPIPE)).
Steps to reproduce
- Install GitLab via omnibus, preferably at v12.6
- Write a cache value with
Rails.cache.write('gitlab:testcache:foo', 'bar', expires_in: 5.hours)via rails console - Attempt to read it with
Rails.cache.read('gitlab:testcache:foo')and get'bar'. - Attempt the same after a moderate delay and get
nil, then immediately again and back to'bar'. - Edit
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/cache/redis_cache_store.rbto removefailsafe :read_entry do - Restart unicorn with
sudo gitlab-ctl restart unicorn - Follow
production.logwithsudo tail --follow /var/log/gitlab/gitlab-rails/production.log. - Note that there are now
Redis::InheritedErrorerrors matching those in #121670 (comment 268387855) - Try
Rails.cache.read('gitlab:testcache:foo')again, this timeRedis::ConnectionError (Connection lost (EPIPE))will be raised wherenilwould have before
Workarounds
Adding reconnect_attempts: 1 to /var/opt/gitlab/gitlab-rails/etc/resque.yml