Rate-limit anonymous requests to Projects List API endpoint
When analysing traffic on Projects List API endpoint, we realised the huge amount of it is anonymous traffic.
Here are the statistics for January 8th:
- total requests: 2,597,483
- requests from anonymous users: 1,063,508 (https://log.gprd.gitlab.net/goto/d329ba00-91c7-11ed-85ed-e7557b0a598c)
- slow requests: 115,429
- slow requests from anonymous users: 37,518 (https://log.gprd.gitlab.net/goto/b1b211b0-91c7-11ed-9f43-e3784d7fe3ca)
Most of the slow queries is using 100 per_page param, making it that harder to produce better times.
I propose rate-limiting this endpoint for anonymous users - heavily. I don't know how to get to the right limit for that, I will ask @reprazent and SREs for advice here.
epic