Simple rate limiter for Gitaly
To protect Gitaly from getting too many backed up requests in its queue, we've added hard limits to the queue size !4270 (merged), and a limit on the queue time !4271 (merged). (see https://gitlab.com/gitlab-org/gitaly/-/issues/4001 for context.)
The second iteration of this is to build a simple rate limiter in Gitaly. (note: suggestion originally from @jacobvosmaer-gitlab)
- Build the simplest rate limiting mechanism we can hope to be effective. E.g. a request counter per project that resets once a minute. A possible refinement would be to give requests different weights according to how much
git
process CPU time they used. - Ship and deploy this iteration but do not enforce limits, just log them. Perhaps also export the maximum observed rate at any time as a prometheus metric.
- Wait for incidents to happen, and see if enforceable patterns emerge.
- Iterate on the non-enforcing rate limiter until you have something useful that does not adversely affect normal user activity.
- Flip the switch to make the rate limiter start enforcing limits.