Log child pipelines that would exceed the per-user pipeline rate limit

What does this MR do and why?

Child pipelines are excluded from the pipeline creation rate limits. That keeps normal fan-out working on the per-commit limit (pipelines_create, 25/min per project, user and commit), but it also means child pipelines don't count toward the per-user limit (pipelines_created_per_user, 2,000/min). The proposed change is to keep them excluded from the per-commit limit and count them toward the per-user one. This MR measures who that would affect before it's enforced.

It adds a log-only experiment behind the ci_log_child_pipeline_rate_limit derisk flag:

  • A shadow rule, pipelines_created_per_user_with_children, copies the per-user limit and uses the Labkit :log action, so it never blocks.
  • Chain::Limit::RateLimit counts every pipeline on the shadow key, child pipelines included. Policy, excluded Duo workflow and lint pipelines are still skipped, sharing one exclusion check with the real limit.
  • When a child pipeline goes over the shadow limit, it logs Child pipeline would exceed the per-user pipeline rate limit with the user, namespace, root namespace, project, commit, parent pipeline and plan. The IDs use the Labkit field names (gl_user_id, gl_project_id, gl_root_namespace_id, ...).
  • The real counters and the pipeline outcome don't change.
Why a shadow key

Calling throttled? on the real key for child pipelines would increment it, using up the user's real allowance before anything is turned on. A separate key gives the combined count without touching the real one.

The shadow rule uses Labkit's :log action. In this code path it doesn't change behaviour: LabkitAdapter.run! returns result.exceeded?, so throttled? returns true once the shadow key is over, the same as with :limit. What keeps this from blocking is that the shadow path only ever logs. :log is there so that:

  • gitlab_labkit_rate_limiter_rule_evaluations_total records it as result="log" rather than result="block", which gives a "would have blocked" count without parsing logs.
  • Labkit reports block? as false, so the rule still can't block if the adapter ever switches to it.
  • It's clear from the rule definition that this is a shadow.

Labkit doesn't log who went over, so the chain step logs that itself. While the flag is on, each pipeline creation makes one extra Redis increment.

References

Screenshots or screen recordings

Not applicable, no UI changes.

How to set up and validate locally

  1. In the Rails console, set a low per-user limit and turn the flag on:

    ApplicationSetting.current.update!(pipeline_limit_per_user: 3)
    Feature.enable(:ci_log_child_pipeline_rate_limit)
  2. In a project, push a .gitlab-ci.yml with a parent job that triggers three or more child pipelines (trigger: include: child.yml).

  3. Run the pipeline. All child pipelines are created.

  4. Check log/application_json.log for Child pipeline would exceed the per-user pipeline rate limit.

Example line from a local GDK run (per-user limit 3, a parent with 5 child pipelines that each trigger a grandchild). All 10 trigger jobs succeeded and 9 lines like this were logged:

{
  "severity": "INFO",
  "time": "2026-11-02T18:21:27.568Z",
  "correlation_id": "b8be926fe5c1062e8a441ab0888cb05f",
  "meta.feature_category": "continuous_integration",
  "meta.caller_id": "Ci::CreateDownstreamPipelineWorker",
  "meta.root_caller_id": "Ci::InitialPipelineProcessWorker",
  "class_name": "Gitlab::Ci::Pipeline::Chain::Limit::RateLimit",
  "gl_namespace_id": 1,
  "gl_root_namespace_id": 1,
  "gl_project_id": 60,
  "gl_user_id": 1,
  "commit_sha": "b033a70fd34962d98ca44aac3792dd6046fe76c7",
  "parent_pipeline_id": 2348,
  "subscription_plan": "default",
  "message": "Child pipeline would exceed the per-user pipeline rate limit"
}

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Max Fan

Merge request reports

Loading
Loading