Skip to content

Simple rate limiter for Gitaly

To protect Gitaly from getting too many backed up requests in its queue, we've added hard limits to the queue size !4270 (merged), and a limit on the queue time !4271 (merged). (see https://gitlab.com/gitlab-org/gitaly/-/issues/4001 for context.)

The second iteration of this is to build a simple rate limiter in Gitaly. (note: suggestion originally from @jacobvosmaer-gitlab)

  1. Build the simplest rate limiting mechanism we can hope to be effective. E.g. a request counter per project that resets once a minute. A possible refinement would be to give requests different weights according to how much git process CPU time they used.
  2. Ship and deploy this iteration but do not enforce limits, just log them. Perhaps also export the maximum observed rate at any time as a prometheus metric.
  3. Wait for incidents to happen, and see if enforceable patterns emerge.
  4. Iterate on the non-enforcing rate limiter until you have something useful that does not adversely affect normal user activity.
  5. Flip the switch to make the rate limiter start enforcing limits.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information