Increase Redis Sentinel maxclients limit on redis-sidekiq
Background
We discovered in production#17468 (comment 1741766338) that we're getting ERR max number of clients reached
responses from Redis Sentinel a few times a day:
We initially thought these were coming from the redis process in redis-sidekiq
(maxclients 50k) but we discovered in #2867 (comment 1774133043) that it is in fact the sentinel process (maxclients 10k).
Impact
This is producing short bursts of user facing errors. While it is not yet impacting SLOs, this is a very worrying development because:
- We are saturating a critical resource, which could get much worse and have cascading effects.
- Our monitoring does not detect this saturation (we have no
redis_exporter
for sentinel). - Our capacity planning did not forecast this saturation.
Proposal
- We should bump maxconns to 50k for Redis Sentinel on ServiceRedisSidekiq (and possibly all other sentinels).
- We should introduce
redis_exporter
for sentinel.
Edited by Igor