Limit the size and number of pipeline variables

What does this MR do and why?

Pipeline variable counts and value sizes are unbounded today. This adds three plan limits, checked when a pipeline is created:

Plan limit GitLab.com target Default
ci_pipeline_variables_count 1200 0 (not set)
ci_variable_value_size 128 KiB 0 (not set)
ci_variable_file_value_size 1 MiB 0 (not set)

A pipeline over a limit fails to be created with an error that names the limit, or the variable for a size limit. Both errors link to the docs. Existing pipelines are not affected. 0 means a limit is not enforced. Administrators set the limits in Admin > Settings > CI/CD or through the plan limits API.

No defaults. Every column defaults to 0, so GitLab Self-Managed and GitLab Dedicated enforce nothing until an administrator sets a value. GitLab.com sets its target values through the plan limits API.

Value size. The kernel caps a single environment variable, name included, at MAX_ARG_STRLEN, which is 128 KiB on systems with 4 KiB pages such as x86_64 and most arm64. A larger value can't reach a job, and the job fails with argument list too long and no hint about the cause. The limit moves that failure to pipeline creation, with an error that names the variable. File type variables are written to a file and only the path goes into the environment, so they get their own limit. The cap applies to every kind of CI/CD variable, so the two size limits are named for all variables. This MR only checks pipeline variables.

Override. The ci_pipeline_variable_limits_override flag (ops, disabled by default) raises the file type value limit to 10 MiB for a top-level namespace. A higher plan limit still applies, and a limit of 0 stays disabled. It does not change the count or the value size limit. The check reads it only after a file type value is over its plan limit. On GitLab.com it is enabled for the namespaces that regularly send larger values.

TRIGGER_PAYLOAD. The trigger API adds TRIGGER_PAYLOAD as a file type variable holding the whole request: the webhook body, variables and inputs. It counts toward the limits like any other variable, so on the trigger API the file type value limit caps the whole request. doc/ci/triggers/_index.md documents this and the alternatives.

The existing 500 million character cap on a value stays as a backstop, and it also applies when a size limit is 0. The 10 MiB total size check in PipelineVariablesArtifactBuilder, behind the ci_enforce_pipeline_variables_total_size_limit flag, is separate and runs after these checks. If a project's plan cannot be resolved, the limits are 0.

The GitLab.com targets were checked against the pipeline creation logs (Rails and Sidekiq) and the stored pipeline variables since February. GitLab team members can see the figures in the internal notes on this merge request and on https://gitlab.com/gitlab-org/gitlab/-/issues/613416.

References

Screenshots or screen recordings

The MR adds three numeric fields to the CI/CD limits section of Admin > Settings > CI/CD, and three rows to the CI/CD limits table on the instance configuration help page.

How to set up and validate locally

  1. In Admin > Settings > CI/CD, set the pipeline variable count limit to 2, the value size limit to 1024, and the file type value size limit to 1024.
  2. Run a pipeline with three variables. Pipeline creation fails with the count error, which links to the docs.
  3. Run a pipeline with one variable whose value is over 1 KiB. Pipeline creation fails, names the variable, and links to the docs.
  4. Run a pipeline with one file type variable whose value is over 1 KiB. Pipeline creation fails the same way.
  5. In a Rails console, run Feature.enable(:ci_pipeline_variable_limits_override, Project.find(<id>).root_namespace). Repeat step 4: the pipeline is created. Repeat steps 2 and 3: they still fail.
  6. Set the limits back to 0 and confirm nothing is enforced.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Oleg Yakovenko

Merge request reports

Loading
Loading