Skip to content

Add per-request limit to parallel remote includes

Furkan Ayhan requested to merge 351250-limit-parallel-count-for-users into master

What does this MR do and why?

After adding the parallel support of remote includes, we needed to limit the number of parallel HTTP requests to avoid abusing IO.

This change adds 2 as the maximum parallel limit per pipeline creation or lint request.

Related to #351250 (closed)

Implementation table: #351250 (comment 1647814064)

Screenshots or screen recordings

  1. Enable the feature flag;
    Feature.enable(:ci_parallel_remote_includes)
  2. Run a basic web server to test slow includes.
    require 'rack'
    
    def run(env)
      puts "URL: #{env['REQUEST_URI']} started at #{Time.now}"
      sleep 5
      puts "URL: #{env['REQUEST_URI']} finished at #{Time.now}"
      "
      test:
        script: echo 'hello world'
      "
    end
    
    app = Proc.new do |env|
      [
        200,
        { "Content-Type" => "text/x-yaml" },
        [run(env)]
      ]
    end
    
    Rack::Server.start(app: app, Port: 9292)
  3. Enable local includes on the Admin page http://gdk.test:3000/admin/application_settings/network Screenshot_2023-06-01_at_09.37.52
  4. Use a basic CI config;
    include:
      - remote: http://0.0.0.0:9292/file1.yml
      - remote: http://0.0.0.0:9292/file2.yml
      - remote: http://0.0.0.0:9292/file3.yml
      - remote: http://0.0.0.0:9292/file4.yml
      - remote: http://0.0.0.0:9292/file5.yml
    
    build:
      script: exit 0
  5. Result; 7ATfHcSbgN

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports