Skip to content

Add CI remote include parallelization

Furkan Ayhan requested to merge 351250-parallelize-ci-remote-includes into master

What does this MR do and why?

Related to #351250 (closed)

In preload_content, we start executing the HTTP call in another thread. Then, when the content method is called, we wait for the HTTP response.

This is the first step. In the next step, we'll work on limiting the max number of parallel HTTP calls in a single-user call.

These changes are behind the FF ci_parallel_remote_includes (#413770 (closed))

How to set up and validate locally

  1. Enable the feature flag;
    Feature.enable(:ci_parallel_remote_includes)
  2. Run a basic web server to test slow includes.
    require 'rack'
    
    def run(env)
      puts "URL: #{env['REQUEST_URI']} started"
      sleep 3
      puts "URL: #{env['REQUEST_URI']} finished"
      "
      test:
        script: echo 'hello world'
      "
    end
    
    app = Proc.new do |env|
      [
        200,
        { "Content-Type" => "text/x-yaml" },
        [run(env)]
      ]
    end
    
    Rack::Server.start(app: app, Port: 9292)
  3. Enable local includes on the Admin page http://gdk.test:3000/admin/application_settings/network Screenshot_2023-06-01_at_09.37.52
  4. Use a basic CI config;
    include:
      - remote: http://0.0.0.0:9292/file1.yml
      - remote: http://0.0.0.0:9292/file2.yml
      - remote: http://0.0.0.0:9292/file3.yml
      - remote: http://0.0.0.0:9292/file4.yml
      - remote: http://0.0.0.0:9292/file5.yml
    
    build:
      script: exit 0
  5. Result; MIerMD5e4W

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports

Loading