Skip to content

Rate limiting improvements for the project archive endpoint (may apply to other endpoints too)

Status

  1. archive_rate_limit is disabled by default
  2. archive_rate_limit is enabled on GitLab.com

Problem to solve

In the GitLab 12.9 release rate limiting for the project archive endpoint was introduced here and enabled by default which broke our builds and someone else's too (see comments on the MR):

!25750 (merged)

This broke a large number of our builds out of the box and it was difficult to track down. After investigating I discovered that disabling all built-in options for rate limiting in GitLab didn't work around the problem. I was preparing to potentially have to rollback the upgrade then discovered the above merge request. I opened the gitlab-rails console and ran:

Feature.disable(:archive_rate_limit)

And now this endpoint is no longer rate limited so we're good for now but this was difficult to debug and doesn't seem to follow the typical pattern other rate limiting in GitLab does.

Intended users

Further details

See proposal.

Proposal

By default archive_rate_limit should be false unless the project archive endpoint respected the other rate limiting settings already present in the admin panel then a default of true would make more sense. In our case, and likely others too, this feature which was appears to have been added for performance and security reasons doesn't really help if unauthenticated requests are not allowed. In other cases, such as ours, we often implement rate limiting outside of GitLab itself before the traffic hits the instance. In this feature's current form however it's either on or off entirely so there's no middle ground for deployments that do want to rate limit this endpoint but actively use it as part of their normal processes too.

In addition, I like to propose some form of documentation improvement for this type of rate limiting. The only information I had to go on to debug this problem were the docs around the rate limiting options in the admin panel and logs like this:

{"severity":"ERROR","time":"2020-03-26T18:10:40.872Z","correlation_id":"<STRING>","message":"Application_Rate_Limiter_Request","env":"project_repositories_archive_request_limit","remote_ip":"<IP_ADDRESS>","request_method":"GET","path":"/api/v4/projects/<NUMBER>/repository/archive.tar.gz?sha=<TAG>","user_id":<USER_ID>,"username":"<USER_NAME>"}

I could be missing something obvious but a Google search for Application_Rate_Limiter_Request and project_repositories_archive_request_limit didn't turn up a lot of helpful information. The only reason I discovered the above MR was because I went diving into the changelog for GitLab and found this line:

After reading over the MR it became apparent that this didn't rate limit didn't work quite like some of the others did for authenticated/unauthenticated requests which explained why disabling all rate limiting didn't solve the problem. So if there's going to be rate limiting applied by GitLab, and it's different from existing rate limiting, it probably should be documented so it's easy to tell where the rate limiting is coming from and how to potentially change or disable it if problems arise. Given the potential operational impact I feel like extra documentation would have been helpful here especially if someone isn't as familiar with GitLab.

Permissions and Security

Requires instance administrator permissions to change. Rate limiting is also a security feature but in some cases may not be warranted. In others, it may impede normal operation.

Documentation

Availability & Testing

TBD

What does success look like, and how can we measure that?

What is the type of buyer?

Starter and above most probably.

Is this a cross-stage feature?

Unknown.

Links / references

Edited by Thong Kuah