Configurable merge train pipeline limits (2/4)

Norman Debaldrequested to merge
gitlab-community/gitlab-org/gitlab:374188-2-backend-logic-and-api into master

What does this MR do and why?

This is MR 2 of 4 in a stacked series that adds configurable merge train pipeline limits, addressing gitlab-org/gitlab#374188.

MR Branch Base Content
1 374188-1-add-max-pipelines-per-merge-train-plan-limit master Database migrations (!228370 (merged))
2 (this MR) 374188-2-backend-logic-and-api 374188-1-add-max-pipelines-per-merge-train-plan-limit Backend logic, REST API, GraphQL, tests
3 374188-3-admin-and-project-ui 374188-2-backend-logic-and-api Admin UI, Project UI, validations (!228380 (merged))
4 374188-4-documentation master Documentation (!228488 (merged))

Currently, merge trains have a hardcoded limit of 20 parallel pipelines (DEFAULT_CONCURRENCY = 20 in ee/app/services/merge_trains/refresh_service.rb). This MR makes the limit configurable at two levels:

  • Plan limit (Iteration 1): Instance-wide, configurable by admins via Plan Limits API and Admin UI. Default: 20.
  • Project limit (Iteration 2): Per-project override via Project API, GraphQL, and Project Settings UI. The effective limit is min(plan_limit, project_limit) when a project limit is set. The plan limit always wins.

Files to review in this MR

Note: This is a stacked MR. The diff includes files from MR 1 (!228370 (merged)). Only the files listed below are new in this MR.

Core logic:

  • ee/app/services/merge_trains/refresh_service.rb - New concurrency_limit method

Plan Limits API (admin-only):

  • lib/api/admin/plan_limits.rb - New optional parameter
  • lib/api/entities/plan_limit.rb - Expose in response
  • app/controllers/admin/plan_limits_controller.rb - Whitelist parameter

Project API:

  • ee/lib/ee/api/helpers/projects_helpers.rb - Parameter + update_params_at_least_one_of
  • ee/lib/ee/api/entities/project.rb - Expose with license check
  • ee/lib/ee/api/projects.rb - License filter in verify_merge_pipelines_attrs!
  • ee/app/controllers/ee/projects_controller.rb - Whitelist parameter
  • ee/app/controllers/ee/projects/settings/merge_requests_controller.rb - Whitelist parameter

GraphQL:

  • ee/app/graphql/ee/types/ci/ci_cd_setting_type.rb - New field
  • ee/app/graphql/ee/mutations/ci/project_ci_cd_settings_update.rb - New argument

Model:

  • ee/app/models/ee/project.rb - Delegate to ci_cd_settings

Tests:

  • ee/spec/services/merge_trains/refresh_service_spec.rb - 5 new scenarios
  • spec/requests/api/admin/plan_limits_spec.rb - GET, PUT, wrong-type tests
  • spec/requests/api/project_attributes.yml - New attribute
  • spec/graphql/types/ci/ci_cd_setting_type_spec.rb - New field
  • ee/spec/models/ee/project_spec.rb - Delegation test
  • spec/models/project_spec.rb - Exclude from CE delegation test

Auto-generated (no review needed):

  • doc/api/graphql/reference/_index.md
  • doc/api/openapi/openapi_v2.yaml
  • doc/api/openapi/openapi_v3.yaml
  • public/-/graphql/introspection_result.json
  • public/-/graphql/introspection_result_no_deprecated.json

License protection

The project-level parameter is protected at multiple layers:

  1. REST API filter (verify_merge_pipelines_attrs!) checks feature_available?(:merge_trains)
  2. Controllers check allow_merge_trains_params? which checks feature_available?(:merge_trains)
  3. Project API entity only exposes the field when :merge_trains is available
  4. The RefreshService is only invoked when a MR is on an active merge train (requires license)

How to set up and validate locally

Plan limit (Admin API)

# Read
curl --header "PRIVATE-TOKEN: <token>" \
  "http://gdk.local:3000/api/v4/application/plan_limits" | jq .max_pipelines_per_merge_train

# Update
curl --request PUT --header "PRIVATE-TOKEN: <token>" \
  "http://gdk.local:3000/api/v4/application/plan_limits" \
  --data "plan_name=default&max_pipelines_per_merge_train=5"

Project limit (Project API)

curl --request PUT --header "PRIVATE-TOKEN: <token>" \
  "http://gdk.local:3000/api/v4/projects/<id>" \
  --data "max_pipelines_per_merge_train=3"

GraphQL

query {
  project(fullPath: "root/my-project") {
    ciCdSettings { maxPipelinesPerMergeTrain }
  }
}

mutation {
  projectCiCdSettingsUpdate(input: {
    fullPath: "root/my-project"
    maxPipelinesPerMergeTrain: 3
  }) {
    ciCdSettings { maxPipelinesPerMergeTrain }
    errors
  }
}

RefreshService logic (Rails console)

project = Project.find(<id>)
project.actual_limits.update!(max_pipelines_per_merge_train: 10)
project.ci_cd_settings.update!(max_pipelines_per_merge_train: 3)

service = MergeTrains::RefreshService.new(project.id, 'main')
service.send(:concurrency_limit)
# => 3 (min of 10 and 3)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

  • Backend logic with tests (5 new scenarios for concurrency limit)
  • Plan Limits API with tests (GET, PUT, wrong-type validation)
  • Project API parameter with license protection
  • GraphQL type and mutation
  • All existing tests continue to pass

Related issues

Part of gitlab-org/gitlab#374188 (parent epic: gitlab-org&2219)

Edited by Norman Debald

Merge request reports