Deploy twemproxy in front of Redis
From https://gitlab.com/gitlab-com/production/issues/431#note_97230109, as the number of Redis connections grows, Redis becomes slower at handling requests and is susceptible to timeouts, especially with Sidekiq. This has led to repeated Redis failovers over the last two weeks and many errors both on the Web and Sidekiq workers.
Currently it looks like when we get close to 6000 connections, things start to behave badly. Reducing the connection load to 5000 by eliminating two Sidekiq besteffort nodes seems to help. Several options also include:
- Tweaking Redis
hzoption - Splitting the Sidekiq Redis cluster from the shared state cluster (this is just a config option but requires some migration work; see https://gitlab.slack.com/archives/CB7P5CJS1/p1534885706000100)
- Reducing the number of connections sidekiq-cluster uses
However, just as we have pgbouncer in front of PostgreSQL, we have gotten to the size where we should consider deploying twemproxy in front of Redis:
- https://blog.twitter.com/developer/en_us/a/2012/twemproxy.html
- http://antirez.com/news/44
- https://tech.trivago.com/2017/01/25/learn-redis-the-hard-way-in-production/
I'll create an Omnibus issue to support this.
Edited by Stan Hu
