Skip to content

Download export parts in parallel

Mehmet Emin INAC requested to merge minac_process_export_parts_concurrently into master

What does this MR do and why?

This MR installs the async gem which will allow us to download the export files concurrently using Fibers.

Other alternatives considered

  1. Using a thread pool: master...minac_use_multiple_threads_to_download_segment_files
  2. Implementing a fiber scheduler: master...minac_implement_non_blocking_file_download

I have decided to install the async gem because;

  1. It's a well-maintained gem.
  2. The thread pool approach does not look that clean.
  3. Implementing a fiber scheduler wasn't that hard as I copied and modified the test one from the Ruby repository but I didn't want to write specs and maintain it.

You can see the benefit of using the non-blocking IO feature of Ruby 3 with fibers by running the following in the rails console;

require 'async'

def test_async
  start_time = Time.now.to_i
  Async do
    100.times do
      Async do
        HTTParty.get("https://httpbin.org/delay/2")
      end
    end
  end

  puts "Finished in #{Time.now.to_i - start_time} seconds"
end

test_async

Related to #440163

Merge request reports