Rate limit Tasks by type requests behind a flag
What does this MR do and why?
Limits the Tasks by type requests (Groups::Analytics::TasksByTypeController#show and #top_labels) to 20 per minute for each user and group. The two actions share one limit. It is behind the gitlab_com_derisk flag tasks_by_type_rate_limit, which is off by default and uses the group's root ancestor as the actor, so it can be rolled back quickly.
Each of these requests can hold a replica connection for up to 15 seconds on large groups, and nothing stops one user from repeating them. The show query can't be made cheaper in PostgreSQL (see the evidence in the issue), and the ClickHouse rebuild was agreed to be a separate project, so the agreed step for Project Bulkhead is a rate limit. No query changes in this MR.
Over the limit, the request gets the application's standard 429 response, and the chart shows its existing generic error alert.
Why 20 per minute
The chart sends two requests on load (top_labels and show). Adam Hegyi checked GitLab.com: user peaks in the last 2-3 days were 6-10 requests per day, and he considered 20 per minute safe (https://gitlab.com/gitlab-org/gitlab/-/work_items/636171#note_4175221559). The direction to do a rate limit now, and ClickHouse separately, is at https://gitlab.com/gitlab-org/gitlab/-/work_items/636171#note_4170198763.
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/636171 (Tasks by type, Project Bulkhead row 6)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/635036 (Project Bulkhead inventory)
- https://gitlab.com/gitlab-org/gitlab/-/work_items/638867 (feature flag rollout issue)
- !268027 (merged) (the
top_labelsquery fix)
Screenshots or screen recordings
Not applicable: there is no UI change.
How to set up and validate locally
- In the Rails console, enable the flag:
Feature.enable(:tasks_by_type_rate_limit). - Open a group's Value stream analytics page with the Tasks by type chart, and check that the chart loads.
- Request
/groups/<group>/-/analytics/type_of_work/tasks_by_type/top_labels?created_after=<date within the last 180 days>&subject=Issuemore than 20 times within a minute while signed in. The 21st request returns 429 withThis endpoint has been requested too many times. Try again later. - Disable the flag with
Feature.disable(:tasks_by_type_rate_limit)and repeat the requests. None is rejected.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.