Default config for runsvdir-start is limiting redis client connections by limiting open file descriptors

Summary

Originating Incident: gitlab-com/gl-infra/production#14497 (closed)

Corrective Action: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23751

The runsvdir-start ulimit settings prevent having a redis service that allows more than 50,000 client connections.

Steps to reproduce

  1. Set the number of redis maxclients connections to greater than 50,000.
  2. Reconfigure via gitlab-ctl reconfigure
  3. Errors are produced: # Server can't set maximum open files to 60032 because of OS error: Operation not permitted.

Here is a comment thread that illustrates the issue when we attempted to set these values higher in GitLab.com production.

What is the current bug behavior?

The upper limit on client connections for a redis server is 50,000 connections.

What is the expected correct behavior?

The value of redis maxclients should be able to be set to a value higher than 50,000.

Relevant logs

Relevant logs

Details of package version

Provide the package version installation details 


$ dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-============================================================
un  gitlab-ce      <none>       <none>       (no description available)
ii  gitlab-ee      15.11.0-ee.0 amd64        GitLab Enterprise Edition (including NGINX, Postgres, Redis)
un  gitlab-fips    <none>       <none>       (no description available)





This is from staging, but it's probably similar to production.

Environment details

  • Operating System: Ubuntu 20.04.3 LTS
  • Installation Target, remove incorrect values:
    • VM: GCP
  • Installation Type, remove incorrect values:
    • Upgrade from version REPLACE-WITH-DETAILS
  • Is there any other software running on the machine: REPLACE-WITH-DETAILS
  • Is this a single or multiple node installation?
  • Resources
    • CPU: Intel(R) Xeon(R) CPU @ 3.10GHz
    • Memory total: 15GB

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`
Edited by Cameron McFarland