Skip to content

Rate limit Gitlab Shell operations

Igor Drozdov requested to merge id-rate-limit-ssh-operations into master

What does this MR do and why?

When a user performs an unusual number of a particular Gitlab Shell operation to a particular project, it makes sense rate limit it and return 429 response code.

The current rate limit is: 600 operations per minute.

I thought ghost1 user would be a problem, but it seems to have different ids: https://log.gprd.gitlab.net/goto/fdd9fc00-7824-11ec-a649-b7cbb8e4f62e

At first, 20 operations per minute sounded like a reasonable option to me, but top 200 project/user perform more than 20 operations per minute (~30k per day). I wonder if the threshold of at least 60 is better in this case: top 40 project/user perform more than 60 operations per minute (~85k per day).

https://log.gprd.gitlab.net/app/lens#/edit/e9d69f90-7826-11ec-a649-b7cbb8e4f62e?_g=h@c823129

The change is behind rate_limit_gitlab_shell feature flag.

Approach

This MR performs rate limiting by user account and project for git operations. Gitlab Shell (Golang) will accept the requests and then make a call to the common Rails rate limiter (backed by Redis). If the user + project limit is exceeds the rate limit then Gitlab shell will then drop further connection requests for that user + project.

The rate limiter is applied at the git command (plumbing) level. Each command has a rate limit of 60/minute - ie: git push has 600 and git pull has another 600 per minute. Because they are using the same plumbing command git-upload-pack, git pull and git clone are in effect the same command.

There is also a rate limiter in place in Gitlay, but the calls will never be made to Gitaly if the rate limit is exceeded in Gitlab Shell + Rails.

The actual rate limit is currently proposed at 20/minute, but this will be discussed further during the maintainer review process.

Edited by Sean Carroll

Merge request reports