Stop reading remote includes past the YAML size limit

What does this MR do and why?

A remote include downloads its full response before it is compared against max_yaml_size_bytes, and with cache: the full body is stored before that check runs. This MR stops reading the response once it passes the limit

  • Passes max_bytes: max_yaml_size_bytes to Gitlab::HTTP for include: remote, so the client stops reading once the response is too large
  • Returns the existing "The provided YAML is too big" include error, without retrying
  • Oversized responses are no longer written to the cache
  • Behind ci_remote_include_max_response_size (gitlab_com_derisk), because today's check runs per YAML document, so a file with a spec: header can be valid while its total size is over the limit
  • Capping the cache: TTL is a follow-up

References

How to set up and validate locally

  1. In the Rails console, run ApplicationSetting.current.update!(max_yaml_size_bytes: 200) and Feature.enable(:ci_remote_include_max_response_size)
  2. Add include: { remote: '<URL of a YAML file larger than 200 bytes>', cache: true } to a project's .gitlab-ci.yml and validate it in the pipeline editor
  3. Confirm the error is "<url>: The provided YAML is too big"
  4. Run bundle exec rspec spec/lib/gitlab/ci/config/external/file/remote_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Avielle

Merge request reports

Loading
Loading