Draft: Implement parallel execution for CI includes
What does this MR do and why?
This MR implements a parallel execution process for CI includes.
- Related to #296532 (closed)
- I used the concurrent-ruby gem, which is installed in our project via a couple of gems. However, we should add this gem individually after making a decision about this MR.
- Documentation for
Concurrent::Promise
: https://ruby-concurrency.github.io/concurrent-ruby/1.1.5/Concurrent/Promise.html - It's behind a feature flag
ci_parallel_include_fetching
.
Screenshots or screen recordings
A simple example;
promises = objects.map do |object|
Concurrent::Promise.new { sleep 5 }.execute
end
results = promises.map do |promise|
promise.wait
if promise.fulfilled?
promise.value
else
raise promise.reason
end
end
Or with the new Module;
Gitlab::Parallel.execute(objects) do |object|
sleep 5
end
How to set up and validate locally
- Create a couple of TCP server to imitate remote files. (to experience a static delay time)
require 'socket'
server = TCPServer.new 5676 # we need a couple of ports
while session = server.accept
request = session.gets
puts request
sleep 5 # delay time
session.print "HTTP/1.1 200\r\n"
session.print "Content-Type: text/html\r\n"
session.print "\r\n"
session.print "Hello world! The time is #{Time.now}"
session.close
end
- Disable
validate_local_request
inlib/gitlab/url_blocker.rb
to be able to use local fake includes. - Open
http://gdk.test:3000/-/graphql-explorer
- Enter this config;
query getCiConfigData($projectPath: ID!, $sha: String, $content: String!) {
ciConfig(projectPath: $projectPath, sha: $sha, content: $content) {
errors
mergedYaml
status
}
}
- With these query variables;
{
"projectPath": "root/project-path",
"sha": "latest sha",
"content": "include:\n - remote: 'http://localhost:5676/a.yml'\n - remote: 'http://localhost:5677/b.yml'\n - remote: 'http://localhost:5678/c.yml'\n - remote: 'http://localhost:5679/d.yml'\n"
}
- Compare the response time with and without
Feature.enable(:ci_parallel_include_fetching)
.
Without ci_parallel_include_fetching
;
With ci_parallel_include_fetching
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.