Sidekiq doesn't fail when disconnected from Redis
As seen in demo 2018-06-01 .
Sidekiq pods were running and reported as healthy. When importing external project, the import was hanging.
In sidekiq logs we could see that it cannot connect to Redis which caused a job to fail. However, sidekiq was still operating correctly. From the sidekiq pod we could get to Redis which suggests it was a temporary failure of some sort.
After killing the pod and getting a new one, the job passed.
We should see if we can have a way to reproduce this and whether we need to create some sort of liveliness probe.