Handle 'too many open files' system error to return proper status code
Under a high load gitaly may operate with too many open files and it is possible that there won't be more file descriptors available for acquisition. In this case almost any RPC returns Internal
or Unknown
status code. That doesn't tell anything about the reason of the failures and the caller continue to make the calls. It makes a failure circle: gitaly fails and returns an error, the caller receives an error and re-attempt the operation which creates more load, so gitaly continues to fail.
In order to prevent getting into the failure circle we need to return special status code that should signal about the problems with resource consumption. So basically we need to return ResourceExhausted
status code in this case.
This should make system more resilient to high loads and help to monitor system state better. That all should result into quicker resolution of the problems like one we had with file descriptors.
/cc @andrashorvath