Add limits configurable through the application settings to rate limit Projects List API
We want to rate limit the Projects List API endpoint. It is heavily used by the anonymous users - which affects our error budget.
We want to use this mechanism: https://gitlab.com/gitlab-org/gitlab/blob/6865b1dae1da1e34b792b2cb81dfbc68033609fe/lib/gitlab/application_rate_limiter.rb#L8
To do it, we need to add application setting to configure it and to allow the self-managed instances to have their own limits.
Implementation
- We will focus on
GET /projects
as it has the most of the anonymous traffic. - This rate limit applies only to requests from anonymous users (so, scope:
request.ip
). This endpoint is heavily used by our paid customers, we don't want to limit that at this moment. - Application limits should be enabled by default.
When implementing application limits
If we are considering enabling or changing a limit, we should do the following (applies to GitLab.com and self-managed):
-
Evaluate if GitLab.com and self-managed should match - Usually, the limits on GitLab.com should be a good match for self-managed but there may be situations in which limits on GitLab.com are not a good match for our self-managed customers. For example, the artifact expiration on GitLab.com was put in place to control costs and this did not apply equally to self-managed customers. -
Evaluate the impact to current users - How many users will be affected by this change? How much of an impact will they feel? If you need help pulling data for GitLab.com, create an issue on the Infrastructure project -
Communicate limits in advance of implementation - Create an issue and facilitate community discussion about the impact the change might have. Raise awareness of the change via social media or a blog post. If the limit will result in a breaking change, do several announcements over a period of time to ensure that everyone has advance notice. -
Communicate the limits in advance to the Quality teams - Quality runs tests against various environments that reuse users and as a result tend to hit limits as a false positive. As a result, Quality needs to be informed to ensure that tests can be adjusted accordingly. -
Proactively notify Customer Success and Support of the change - Reach out in #customer-success and #support_escalations to announce the upcoming change, and consider discussing in the next All CS Team Call to solicit feedback. -
Ensure Customer Success and Support are equipped to help users - Make sure that Customer Success and Support has access to the documentation that they need to help customers who contact them regarding the limit. -
Document the limits on docs.gitlab.com -
Make sure that the limit is documented on the page for the feature and include details such as if it's configurable, what the default value is, and what impact this can have on the end user. -
Document the limit for customers on the instance limits help page, ensuring the limit for gitlab.com is specified. Include instructions on how the limit can be changed on self-managed instances. -
If the limit is time based, link to that section from the Rate limits page -
Communicate the limits in the release post - When the limit is rolled out, make sure to document this change in the next release post. -
Communicate directly to affected users - Especially if the limit is going to have a significant impact to users, consider reaching out directly to notify those users of the change, and any available remedies, workarounds, or best practices that may help mitigate that impact. To send out an email to affected users, work with Support to create an email request.
Availability & Testing
- No new E2E tests or updates needed at this time, since our requests to this endpoint are authenticated
- Since this issue is only adding the application setting, notes around testing the rate limit itself will be included in #388437 (closed)
Edited by Christina Lohr