Limit the size and number of pipeline variables

What does this MR do and why?

Pipeline variable counts and value sizes are unbounded today. This adds three plan limits, checked when a pipeline is created:

Plan limit	GitLab.com	Default (GitLab Self-Managed and GitLab Dedicated)
ci_pipeline_variables_count	1200	3600
ci_variable_value_size	120 KiB	360 KiB (368640 bytes)
ci_variable_file_value_size	1 MiB	3 MiB (3145728 bytes)

A pipeline over a limit fails to be created with an error that names the limit, or the variable for a size limit. Existing pipelines are not affected. Administrators can change each limit in Admin > Settings > CI/CD or through the plan limits API, and 0 turns a limit off.

Value size. The kernel caps a single environment variable, name included, at MAX_ARG_STRLEN, which is 128 KiB on systems with 4 KiB pages such as x86_64 and most arm64. A value near that can't reach a job, and the job fails with argument list too long and no hint about the cause. 120 KiB leaves some headroom. File type variables are written to a file and only the path goes into the environment, so they get their own, higher limit. The cap applies to every kind of CI/CD variable, so the two size limits are named for all variables. This MR only checks pipeline variables.

Defaults. The defaults are three times the GitLab.com values. Self-Managed and Dedicated enforce them on upgrade with no usage data from those instances, so they get headroom. For value size this also leaves room for runners without the 128 KiB cap, such as macOS or kernels with 64 KiB pages.

GitLab.com starts at 0. A data migration sets all three limits to 0 on every plan except default. On GitLab.com every namespace resolves to one of those plans, because EE::Namespace#actual_plan only returns Plan.default off SaaS, so nothing is enforced there until the values are set through the plan limits API. Self-Managed and Dedicated only use default, so they enforce the column defaults. The migration's down is empty, so a rollback does not start enforcing on GitLab.com or overwrite values set since.

There is no feature flag. Setting a limit to 0 stops enforcing it without a deployment. A limit applies to a whole plan, so it cannot be lifted for a single namespace. If a plan cannot be resolved, the limit is 0.

The existing 500 million character cap on a value stays as a backstop, and it also applies when a size limit is 0. The 10 MiB total size check in PipelineVariablesArtifactBuilder, behind the ci_enforce_pipeline_variables_total_size_limit flag, is separate and runs after these checks.

The GitLab.com values were checked against data from the pipeline creation logger added in !253383 (merged). GitLab team members can see the figures in the internal notes on this merge request. !262110 (merged) lowers the logger's value threshold to 120 KiB and records the size per variable type.

TRIGGER_PAYLOAD, why it is affected and why that is acceptable

Ci::PipelineTriggerService adds TRIGGER_PAYLOAD to every trigger API pipeline, as a file type variable holding a JSON dump of the request params. Two things made this look risky: a large webhook body fails pipeline creation even though GitLab generated the value, and params includes variables, so a caller's variables are stored twice and the copy inside the payload is measured as one value.

Both cases need roughly a megabyte in a single trigger call. Observed value sizes are well below that, so this is theoretical for now.

Separately, TRIGGER_PAYLOAD should not be a pipeline variable at all. It is documented as a predefined variable, yet unlike every other one it is stored in the user's variable namespace, which is why Chain::Build::Associations has to exempt the key from the permission check, and why a caller passing their own TRIGGER_PAYLOAD gets a duplicate key error. Out of scope here.

References

• https://gitlab.com/gitlab-org/gitlab/-/issues/613416 (confidential)
• https://gitlab.com/gitlab-org/gitlab/-/work_items/613417, the 120 KiB value for job variables (confidential)
• !253383 (merged), merged, logs the count and largest value size
• !262110 (merged), logs values from 120 KiB and the size per variable type
• Logs for pipelines with 1000 or more variables
• Lens visualization for the pipeline variable max bytesize
• Lens visualization for the pipeline variables max count

Screenshots or screen recordings

The MR adds three numeric fields to the CI/CD limits section of Admin > Settings > CI/CD, and three rows to the CI/CD limits table on the instance configuration help page.

How to set up and validate locally

1. Run a pipeline with a variable whose value is over 360 KiB. Pipeline creation fails and names the variable.
2. Run a pipeline with more than 3600 variables. Pipeline creation fails with the count error.
3. Change a limit in Admin > Settings > CI/CD, or through the plan limits API, and confirm the new value applies. Set it to 0 and confirm the limit no longer applies.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.