Skip to content

Silent Redis::InheritedError on unicorn fork leads to nil cache reads via Redis::ConnectionError (EPIPE)

Problem

Rails.cache silences Redis errors leading to misleading nil cache reads, causing problems such as #121670 (closed). In particular we are silently hiding Redis::InheritedError (Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.) as well as Redis::ConnectionError (Connection lost (EPIPE)).

Steps to reproduce

  1. Install GitLab via omnibus, preferably at v12.6
  2. Write a cache value with Rails.cache.write('gitlab:testcache:foo', 'bar', expires_in: 5.hours) via rails console
  3. Attempt to read it with Rails.cache.read('gitlab:testcache:foo') and get 'bar'.
  4. Attempt the same after a moderate delay and get nil, then immediately again and back to 'bar'.
  5. Edit /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/cache/redis_cache_store.rb to remove failsafe :read_entry do
  6. Restart unicorn with sudo gitlab-ctl restart unicorn
  7. Follow production.log with sudo tail --follow /var/log/gitlab/gitlab-rails/production.log.
  8. Note that there are now Redis::InheritedError errors matching those in #121670 (comment 268387855)
  9. Try Rails.cache.read('gitlab:testcache:foo') again, this time Redis::ConnectionError (Connection lost (EPIPE)) will be raised where nil would have before

Workarounds

Adding reconnect_attempts: 1 to /var/opt/gitlab/gitlab-rails/etc/resque.yml