Limit the size of interpolated CI config

What does this MR do and why?

Interpolation checked the size of each string node before it replaced the blocks, not the result. One input value referenced many times could build hundreds of megabytes in memory. The only later check is the per-job config limit, which runs when the pipeline is saved, after the value is already built.

This MR predicts the size of each string node before it builds it. One running total is shared across all files of a pipeline, through External::Context#mutate, the same way as filter_state.

  • Above max_yaml_size_bytes (2 MB by default), the total is logged once per pipeline: CI config interpolation result is larger than max_yaml_size_bytes, with total_bytes. This ships without a flag.
  • Above 10 times max_yaml_size_bytes, interpolation fails with interpolated config is too large, behind ci_limit_interpolated_config_size.

A whole-node block (key: $[[ inputs.x ]]) counts the size of its value too. Array inputs are usually used as the whole node and have no size limit of their own, so script: $[[ inputs.list ]] in many jobs must count toward the total.

With the flag on, the first node that does not fit is logged with CI config interpolation was stopped at the failure limit, so the rollout shows how often the limit fails a pipeline.

References

Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/632683

Rollout issue: #638429

How to set up and validate locally

bin/rspec spec/lib/gitlab/ci/config/interpolation/template_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Laura Montemayor

Merge request reports

Loading
Loading