Corrective action: Increase open file limit for redis.

Summary

During production#14497 (closed) we attempted to increase the redis-cache maxclients setting to 60k. However this setting depends on the Max open file limit to match.

Currently the redis process gets it's limits set by the /opt/gitlab/embedded/bin/runsvdir-start binary, which is called by the systemd gitlab-runsvdir service.

In order to bump the maxclients setting we need to be able to modify and increase the limit being set by the ulimit -n 50000 line on the runsvdir-start script.

We proved this by manually modifying the script on a redis-cache secondary instance in gstg. See details at production#14497 (comment 1400579323)

Related Incident(s)

Originating issue(s): gitlab-com/gl-infra/production#ISSUE_ID

Desired Outcome/Acceptance Criteria

• Write a Chef recipe that allows to override the Max Open Files setting for the redis instance, by modifying the limit value on /opt/gitlab/embedded/bin/runsvdir-start.
• Set an attribute override on the gprd-base-db-redis-server-cache.json and increase the Max Open Files to 60k.
• Apply the changes on the redis-cache instances on GPRD.

Associated Services

Corrective Action Issue Checklist

• Link the incident(s) this corrective action arose out of
• Give context for what problem this corrective action is trying to prevent from re-occurring
• Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
• Assign a priority (this will default to 'Reliability::P4')