Backend: Add ci_max_total_yaml_size_bytes as application setting
Background
We want to increase the number of CI includes, however, we need to solve some performance problems first. One of the problems is that if we allow an unlimited number of includes, then we'll have memory problems with the included files.
To prevent this, we want to introduce a new application setting to limit the max YAML size of total files; ci_max_total_yaml_size_bytes
. We'll use this in either lib/gitlab/ci/config.rb
or lib/gitlab/ci/config/external/context.rb
.
Helpful information
from https://gitlab.com/gitlab-org/gitlab/-/issues/370392#note_1308847292
Today, we allow a file can be max
max_yaml_size_bytes
when extended. By default, it's 1MB per file. And since I don't see anyGitLab.com
specific limit, I guess we are also using the default. And, since we currently allow 150 includes, we allow max 150 MB per pipeline. So, the defaultMAX_SIZE
should probably be 150 MB to avoid breaking changes.Note; 'max_yaml_size_bytes' is configurable by instance: https://docs.gitlab.com/ee/administration/instance_limits.html#maximum-size-and-depth-of-cicd-configuration-yaml-files.
Update: However, we should calculate this dynamically in a migration; max_yaml_size_bytes * ci_max_includes
.
Proposal
Set this new ci_max_total_yaml_size_bytes
as an instance setting. that default to max_yaml_size_bytes * ci_max_includes
. And enforce this new limit (ci_max_total_yaml_size_bytes
) as the maximum total size of a pipeline config as the pipeline configs are evaluated.
Documentation
Settings needs to be documented. https://docs.gitlab.com/ee/administration/configure.html
MR addressing the issue.