Add CI remote include parallelization

What does this MR do and why?

Related to #351250 (closed)

In preload_content, we start executing the HTTP call in another thread. Then, when the content method is called, we wait for the HTTP response.

This is the first step. In the next step, we'll work on limiting the max number of parallel HTTP calls in a single-user call.

These changes are behind the FF ci_parallel_remote_includes (#413770 (closed))

How to set up and validate locally

1. Enable the feature flag;

   Feature.enable(:ci_parallel_remote_includes)

2. Run a basic web server to test slow includes.

   require 'rack'
   
   def run(env)
     puts "URL: #{env['REQUEST_URI']} started"
     sleep 3
     puts "URL: #{env['REQUEST_URI']} finished"
     "
     test:
       script: echo 'hello world'
     "
   end
   
   app = Proc.new do |env|
     [
       200,
       { "Content-Type" => "text/x-yaml" },
       [run(env)]
     ]
   end
   
   Rack::Server.start(app: app, Port: 9292)

3. Enable local includes on the Admin page http://gdk.test:3000/admin/application_settings/network
4. Use a basic CI config;

   include:
     - remote: http://0.0.0.0:9292/file1.yml
     - remote: http://0.0.0.0:9292/file2.yml
     - remote: http://0.0.0.0:9292/file3.yml
     - remote: http://0.0.0.0:9292/file4.yml
     - remote: http://0.0.0.0:9292/file5.yml
   
   build:
     script: exit 0

5. Result;

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

changed milestone to %16.7

added CI lint Category:Pipeline Composition Error Budget Improvement backend bugperformance candidate16.7 customer customer priority10 devopsverify grouppipeline authoring maintenanceperformance missed-deliverable missed:16.1 missed:16.2 missed:16.4 sectionci severity3 typemaintenance workflowblocked labels

assigned to @furkanayhan

added typebug label and removed maintenanceperformance typemaintenance labels

mentioned in issue #413770 (closed)

added feature flag feature flagexists labels

	1 Message
	CHANGELOG missing: If this merge request needs a changelog entry, add the Changelog trailer to the commit message you want to add to the changelog. If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	@wandering_person (UTC+8, 7 hours ahead of author)	@pedropombeiro (UTC+1, same timezone as author)
~"Verify"	Reviewer review is optional for ~"Verify"	@drew (UTC+0, 1 hour behind author)

Please check reviewer's status!

• Reviewer is available!
• Reviewer is unavailable!

Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@9308e3bb

Allure report

allure-report-publisher generated test report!

e2e-test-on-gdk: test report for be08ea0e

expand test summary

+-----------------------------------------------------------------------+
|                            suites summary                             |
+------------------+--------+--------+---------+-------+-------+--------+
|                  | passed | failed | skipped | flaky | total | result |
+------------------+--------+--------+---------+-------+-------+--------+
| Create           | 40     | 0      | 7       | 0     | 47    | ✅     |
| Manage           | 0      | 0      | 1       | 0     | 1     | ➖     |
| Plan             | 55     | 0      | 0       | 0     | 55    | ✅     |
| Govern           | 57     | 0      | 0       | 0     | 57    | ✅     |
| Monitor          | 8      | 0      | 0       | 0     | 8     | ✅     |
| Package          | 0      | 0      | 1       | 0     | 1     | ➖     |
| Data Stores      | 22     | 0      | 0       | 0     | 22    | ✅     |
| Framework sanity | 0      | 0      | 1       | 0     | 1     | ➖     |
| Verify           | 32     | 0      | 0       | 0     | 32    | ✅     |
+------------------+--------+--------+---------+-------+-------+--------+
| Total            | 214    | 0      | 10      | 0     | 224   | ✅     |
+------------------+--------+--------+---------+-------+-------+--------+

e2e-package-and-test: test report for be08ea0e

expand test summary

+-------------------------------------------------------------+
|                       suites summary                        |
+--------+--------+--------+---------+-------+-------+--------+
|        | passed | failed | skipped | flaky | total | result |
+--------+--------+--------+---------+-------+-------+--------+
| Verify | 92     | 0      | 18      | 0     | 110   | ✅     |
+--------+--------+--------+---------+-------+-------+--------+
| Total  | 92     | 0      | 18      | 0     | 110   | ✅     |
+--------+--------+--------+---------+-------+-------+--------+

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@1de9c917

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@f60002e4

added 1 commit

• 5c227ed9 - Add FF info and fix a Rubocop failure

Compare with previous version

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@e26091d5

changed the description

marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

resolved all threads

@ayufan And, here is the result of all the work done

In the next step, we'll work on limiting the max number of parallel HTTP calls in a single-user call because right now a single pipeline lint/create request can try to send 150 parallel HTTP requests.

requested review from @ayufan

@furkanayhan Looks great. Take a look at suggestion.

mentioned in issue #351250 (closed)

removed review request for @ayufan

requested review from @ayufan

removed workflowblocked label

added 712 commits

• 5c227ed9...01a740a2 - 709 commits from branch master
• 4dbba762 - Add CI remote include parallelization
• 8ceab286 - Add FF info and fix a Rubocop failure
• c502a029 - Remove metric config_file_fetch_remote_content

Compare with previous version

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@853a9603

added 247 commits

• c502a029...f9fbb409 - 243 commits from branch master
• 7cabc3f6 - Add CI remote include parallelization
• 9b7e65fd - Add FF info and fix a Rubocop failure
• 3ef176f1 - Remove metric config_file_fetch_remote_content
• 77c842a9 - Remove preload_content from content

Compare with previous version

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@66a9d778

added 177 commits

• 77c842a9...e781a808 - 172 commits from branch master
• faf2428c - Add CI remote include parallelization
• 0da6e5e9 - Add FF info and fix a Rubocop failure
• 304d2220 - Remove metric config_file_fetch_remote_content
• 3731f0a5 - Remove preload_content from content
• be08ea0e - Use the one-ff strategy

Compare with previous version

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@1658d998

resolved all threads

approved this merge request

@ayufan, thanks for approving this merge request.

This is the first time the merge request has been approved. To ensure we don't only run predictive pipelines, and we don't break master, a new pipeline will be started shortly.

Please wait for the pipeline to start before resolving this discussion and set auto-merge for the new pipeline. See merging a merge request for more details.

added pipeline:mr-approved label

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@2ad04269

resolved all threads

enabled an automatic merge when the pipeline for 92e5212c succeeds

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@92e5212c

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@d714a2ab

mentioned in commit gitlab-org-sandbox/gitlab-jh-validation@4a1449bb

merged

mentioned in commit d3f4ac1a

added workflowstaging-canary label

@furkanayhan This merge request was deployed to the workflowstaging-canary environment. You may want to enable the associated feature flag on this environment with /chatops run feature set ci_parallel_remote_includes true --staging.

This message was generated automatically. You're welcome to improve it.

@furkanayhan This merge request was deployed to the workflowcanary environment. You may want to enable the associated feature flag on this environment with /chatops run feature set ci_parallel_remote_includes true --production.

This message was generated automatically. You're welcome to improve it.

@furkanayhan This merge request was deployed to the workflowstaging environment. You may want to enable the associated feature flag on this environment with /chatops run feature set ci_parallel_remote_includes true --staging.

This message was generated automatically. You're welcome to improve it.

@furkanayhan This merge request was deployed to the workflowproduction environment. You may want to enable the associated feature flag on this environment with /chatops run feature set ci_parallel_remote_includes true --production.

This message was generated automatically. You're welcome to improve it.

added workflowcanary label and removed workflowstaging-canary label