Handle Gitaly limit errors in GitLab Rails
One cause of production incidents is when a surge of traffic can land in Gitaly where it tries to serve the requests. However, some of these requests end up needing resource intensive operations. https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6172 is an example.
To prevent Gitaly from falling over in these scenarios, the producer of the traffic ought to be notified and limited. Gitaly already has this capability through rate limiting and concurrency limiting.
Gitaly will return an error message when requests hit a certain limit. This way, Gitaly won't enter into a resource exhaustion failure mode.
This error message is protobuf message defined by gitaly!4476 (merged).
Once Rails receives this message, it needs to propagate this error in a helpful way to both API callers (and return an HTTP 429), and the application (and display a helpful error banner)