Corrective action: Increase open file limit for redis.
Summary
During production#14497 (closed) we attempted to increase the redis-cache
maxclients setting to 60k. However this setting depends on the Max open file
limit to match.
Currently the redis process gets it's limits set by the /opt/gitlab/embedded/bin/runsvdir-start
binary, which is called by the systemd gitlab-runsvdir
service.
In order to bump the maxclients
setting we need to be able to modify and increase the limit being set by the ulimit -n 50000
line on the runsvdir-start script.
We proved this by manually modifying the script on a redis-cache secondary instance in gstg. See details at production#14497 (comment 1400579323)
Related Incident(s)
Originating issue(s): gitlab-com/gl-infra/production#ISSUE_ID
Desired Outcome/Acceptance Criteria
- Write a Chef recipe that allows to override the
Max Open Files
setting for the redis instance, by modifying the limit value on/opt/gitlab/embedded/bin/runsvdir-start
. - Set an attribute override on the gprd-base-db-redis-server-cache.json and increase the Max Open Files to 60k.
- Apply the changes on the
redis-cache
instances on GPRD.
Associated Services
Corrective Action Issue Checklist
-
Link the incident(s) this corrective action arose out of -
Give context for what problem this corrective action is trying to prevent from re-occurring -
Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
Assign a priority (this will default to 'Reliability::P4')