Default config for runsvdir-start is limiting redis client connections by limiting open file descriptors
Summary
Originating Incident: gitlab-com/gl-infra/production#14497 (closed)
Corrective Action: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23751
The runsvdir-start ulimit settings prevent having a redis service that allows more than 50,000 client connections.
Steps to reproduce
- Set the number of redis
maxclients
connections to greater than 50,000. - Reconfigure via
gitlab-ctl reconfigure
- Errors are produced:
# Server can't set maximum open files to 60032 because of OS error: Operation not permitted.
Here is a comment thread that illustrates the issue when we attempted to set these values higher in GitLab.com production.
What is the current bug behavior?
The upper limit on client connections for a redis server is 50,000 connections.
What is the expected correct behavior?
The value of redis maxclients
should be able to be set to a value higher than 50,000.
Relevant logs
Relevant logs
Details of package version
Provide the package version installation details
$ dpkg-query -l "gitlab-*" Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==============-============-============-============================================================ un gitlab-ce <none> <none> (no description available) ii gitlab-ee 15.11.0-ee.0 amd64 GitLab Enterprise Edition (including NGINX, Postgres, Redis) un gitlab-fips <none> <none> (no description available)
This is from staging, but it's probably similar to production.
Environment details
- Operating System:
Ubuntu 20.04.3 LTS
- Installation Target, remove incorrect values:
- VM: GCP
- Installation Type, remove incorrect values:
- Upgrade from version
REPLACE-WITH-DETAILS
- Upgrade from version
- Is there any other software running on the machine:
REPLACE-WITH-DETAILS
- Is this a single or multiple node installation?
- Resources
- CPU:
Intel(R) Xeon(R) CPU @ 3.10GHz
- Memory total:
15GB
- CPU:
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
Edited by Cameron McFarland