Determine and set timeout for Redis instances in our infrastructure
During recent incidents (here and here) with ServiceRedisClusterRateLimiting we saw a spike in incoming client connections to Redis. Meanwhile suspectedly existing clients connections became stale, but redis server kept open the client connections until timeout
(Currently set to 20 minutes) expired.
This issue is to re-evaluate the timeout value for our Redis instances and update this configuration.
Goals
-
Determine an appropriate value for timeout
and determine if this also affects replica connections👉 #192 (comment 2534872646) -
Apply updated timeout
inpre
👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20022 -
Apply updated timeout
ingstg
👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20023 -
Apply updated timeout
ingprd
👉 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20027
Edited by Furhan Shabir