Pulling artifacts is very slow with high latency connections
Background
Today I built gnome-build-meta and was a bit astounded at how long it took to download the base bootstrap system image:
[01:48:20][41fd1dbc][ pull:bootstrap-junction.bst:bootstrap-platform-extract.bst] SUCCESS freedesktop-sdk-bootstrap/bootstrap-platform-extract/41fd1dbc-pull.12566.logThe big base image artifact is not the only one that is very slow, after two hours, I'm still downloading and have not started to build anything yet.
I'm betting that this can be reproduced with the following steps:
- Be somewhere outside of Europe, like Korea or Canada or Brazil
- Try to build something from
gnome-build-meta
I suspect that we are making a lot of small requests when downloading, causing higher latency connections with more than sufficient bandwidth, to be very slow.
Task description
I think that we should be able to stream the objects over a single request when pulling the artifacts, after having determined which objects will be required for download.
Acceptance Criteria
The performance of downloading an artifact should be similar to the performance of downloading a single file of the same size from the same network.