Limit the size of interpolated CI config
What does this MR do and why?
Interpolation checked the size of each string node before it replaced the blocks, not the result. One input value referenced many times could build hundreds of megabytes in memory. The only later check is the per-job config limit, which runs when the pipeline is saved, after the value is already built.
This MR predicts the size of each string node before it builds it. One running total is shared across all files of a pipeline, through External::Context#mutate, the same way as filter_state.
- Above
max_yaml_size_bytes(2 MB by default), the total is logged once per pipeline:CI config interpolation result is larger than max_yaml_size_bytes, withtotal_bytes. This ships without a flag. - Above 10 times
max_yaml_size_bytes, interpolation fails withinterpolated config is too large, behindci_limit_interpolated_config_size.
A whole-node block (key: $[[ inputs.x ]]) counts the size of its value too. Array inputs are usually used as the whole node and have no size limit of their own, so script: $[[ inputs.list ]] in many jobs must count toward the total.
With the flag on, the first node that does not fit is logged with CI config interpolation was stopped at the failure limit, so the rollout shows how often the limit fails a pipeline.
References
Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/632683
Rollout issue: #638429
How to set up and validate locally
bin/rspec spec/lib/gitlab/ci/config/interpolation/template_spec.rbMR acceptance checklist
Evaluate this MR against the MR acceptance checklist.