Add rate limit for pipeline cancellation

Daniel Prauserequested to merge
627270-rate-limit-pipeline-cancel into master

What does this MR do and why?

Adds a rate limit for user-triggered pipeline cancellation. Cancelling a pipeline finishes every cancelable job in its tree, so the cost of a cancel scales with the size of the pipeline. Without a limit, repeated cancels can put excessive load on the instance.

Two limits are introduced, both on a rolling one-minute window:

  • Per user + pipeline: fixed at 5/min, not configurable.
  • Per user + project: configurable via the application setting pipeline_cancel_limit_per_user_project, defaulting to 120/min. Setting it to 0 disables the per-project limit (the per-pipeline limit still applies).

The rate limit applies to the REST cancel endpoint, the pipelineCancel GraphQL mutation, and the UI (which uses these same entry points). It does not apply to automatic cancellations, such as auto_cancel_on_job_failure: Ci::UserCancelPipelineWorker opts out via rate_limit: false.

The check is implemented in Ci::CancelPipelineService and runs before the authorization check, so throttled calls (including unauthorized ones) still count against the bucket. It is skipped for internal callers and when there is no current user.

When a caller is throttled:

  • REST returns 429 Too Many Requests.
  • GraphQL returns the error in the mutation's errors array.
  • Message in both cases: "This endpoint has been requested too many times. Try again later."

The whole feature is guarded by the rate_limit_pipeline_cancel feature flag (type gitlab_com_derisk), disabled by default, targeted for milestone 19.4.

This mirrors the existing pipeline-retry rate limit (!254367 (merged)) for consistency between the two operations.

Related issue: gitlab-org/gitlab#627270 Rollout issue: #628993 (closed)

Screenshots or screen recordings

N/A — backend rate limit, no UI changes.

How to set up and validate locally

  1. In a rails console, enable the feature flag: 
    Feature.enable(:rate_limit_pipeline_cancel)
  2. Optionally lower the per-project limit to make throttling easy to trigger: 
    ApplicationSetting.current.update!(pipeline_cancel_limit_per_user_project: 1)
  3. Cancel the same pipeline via the API twice in quick succession, e.g.: 
    POST /api/v4/projects/:id/pipelines/:pipeline_id/cancel
  4. The first call should succeed; the second call should return 429 Too Many Requests with the rate limit message.

MR acceptance checklist

  • Tests are added for both rate limits (per user + pipeline, per user + project) and for the disable-via-0 behavior.
  • Documentation is updated to describe the new pipeline_cancel_limit_per_user_project application setting and the fixed per-pipeline limit.
  • A feature flag entry (rate_limit_pipeline_cancel, type gitlab_com_derisk) is included, disabled by default.
  • The change is fully behind the disabled-by-default feature flag; no behavior change when the flag is off.

Merge request reports