Skip to content

Draft: Implement parallel execution for CI includes

Furkan Ayhan requested to merge 296532-parallel-http-calls-for-ci-includes into master

What does this MR do and why?

This MR implements a parallel execution process for CI includes.

Screenshots or screen recordings

A simple example;

promises = objects.map do |object|
  Concurrent::Promise.new { sleep 5 }.execute
end

results = promises.map do |promise|
  promise.wait

  if promise.fulfilled?
    promise.value
  else
    raise promise.reason
  end
end

Or with the new Module;

Gitlab::Parallel.execute(objects) do |object|
  sleep 5
end

How to set up and validate locally

  1. Create a couple of TCP server to imitate remote files. (to experience a static delay time)
require 'socket'

server = TCPServer.new 5676 # we need a couple of ports

while session = server.accept
  request = session.gets
  puts request

  sleep 5 # delay time

  session.print "HTTP/1.1 200\r\n"
  session.print "Content-Type: text/html\r\n"
  session.print "\r\n"
  session.print "Hello world! The time is #{Time.now}"

  session.close
end
  1. Disable validate_local_request in lib/gitlab/url_blocker.rb to be able to use local fake includes.
  2. Open http://gdk.test:3000/-/graphql-explorer
  3. Enter this config;
query getCiConfigData($projectPath: ID!, $sha: String, $content: String!) {
  ciConfig(projectPath: $projectPath, sha: $sha, content: $content) {
    errors
    mergedYaml
    status
  }
}
  1. With these query variables;
{
  "projectPath": "root/project-path",
  "sha": "latest sha",
  "content": "include:\n  - remote: 'http://localhost:5676/a.yml'\n  - remote: 'http://localhost:5677/b.yml'\n  - remote: 'http://localhost:5678/c.yml'\n  - remote: 'http://localhost:5679/d.yml'\n"
}
  1. Compare the response time with and without Feature.enable(:ci_parallel_include_fetching).

Without ci_parallel_include_fetching;

Screen_Shot_2021-10-01_at_13.23.31

With ci_parallel_include_fetching

Screen_Shot_2021-10-01_at_13.19.26

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports