Add rate limit to Pipeline Deletion
What does this MR do and why?
Pipeline deletion has no rate limit today. A bad actor or a buggy script can delete pipelines fast enough to strain the system. Deleting a live pipeline is expensive: every job is cancelled synchronously and child pipelines cascade, so one call can cost up to about 1,000 database queries and 26 seconds.
This MR adds two per-user rate limits to pipeline deletion, behind the default-off feature flag rate_limit_pipeline_delete (milestone 19.5). Both reset every minute.
- Per user and pipeline: fixed at 5 per minute, not configurable.
- Per user and project: new application setting
pipeline_delete_limit_per_user_project, default 400. Setting it to 0 removes this limit. The default comes from a 24 hour production sample: 205 of 206 actors stayed under 400 per minute, and the one exception was a short burst.
The check lives in Ci::DestroyPipelineService#execute, so it covers the REST delete endpoint, the pipelineDestroy GraphQL mutation, and the UI. unsafe_execute is not throttled, so automatic pipeline cleanup, project deletion, and batch issuable deletion are unaffected.
Two details for reviewers: the per-pipeline check runs first and short-circuits, so repeated deletes of one pipeline never eat into the per-project budget. The counter increments before the permission check, since a rejected call still costs work, but the permission error is raised first so the user sees the real reason.
When throttled: REST returns 429 Too Many Requests, GraphQL returns the error in errors, and the UI shows an error without deleting. Message: This endpoint has been requested too many times. Try again later.
This also adds the "Maximum pipeline deletion rate" field under Admin > Settings > CI/CD, the matching REST settings API parameter, docs, and specs.
While this branch was open, master gained three sibling limits: pipeline cancel, pipeline retry, and job retry/manual job run. They are independent and sit side by side on purpose, so the settings list, admin form, rule registry, translations, and docs show all of them together. That is not a merge artifact.
References
- Feature issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/627268
- Feature flag rollout issue: #631179
- Rate-limit endpoint audit (WS9): https://gitlab.com/gitlab-org/gitlab/-/issues/605323
How to set up and validate locally
- In the Rails console:
Feature.enable(:rate_limit_pipeline_delete) - In the Rails console:
Gitlab::CurrentSettings.current_application_settings.update!(pipeline_delete_limit_per_user_project: 2) - Delete three pipelines in one project within a minute, via UI, REST, or
pipelineDestroy. The third is blocked: 429 from REST, anerrorsentry from GraphQL, an error banner in the UI. - Check the admin field round-trips under Admin > Settings > CI/CD, and that setting it to 0 removes the per-project limit.
- Confirm that deleting a project or running automatic pipeline cleanup is not throttled.
The fixed per-pipeline limit is hard to trigger by hand, since delete is idempotent. It is covered in spec/services/ci/destroy_pipeline_service_spec.rb.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.