Skip to content

Deploy twemproxy in front of Redis

From https://gitlab.com/gitlab-com/production/issues/431#note_97230109, as the number of Redis connections grows, Redis becomes slower at handling requests and is susceptible to timeouts, especially with Sidekiq. This has led to repeated Redis failovers over the last two weeks and many errors both on the Web and Sidekiq workers.

Currently it looks like when we get close to 6000 connections, things start to behave badly. Reducing the connection load to 5000 by eliminating two Sidekiq besteffort nodes seems to help. Several options also include:

However, just as we have pgbouncer in front of PostgreSQL, we have gotten to the size where we should consider deploying twemproxy in front of Redis:

image

I'll create an Omnibus issue to support this.

Edited by Stan Hu