Slow downloads with ostree plugin compared to vanilla flatpak/ostree

Summary

When BuildStream has to download a flatpak runtime it takes much more time than flatpak or ostree command.

It do not recover from a interrump (Ctrl+C) and retrying starts from beginning

Steps to reproduce

Clean all cache from flatpak

sudo rm -rf /var/lib/flatpak/
sudo rm -rf ~/.local/share/flatpak/

Download from flatpak:

flatpak --user remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
flatpak --user install flathub org.gnome.Sdk//3.28

Installing for user: org.gnome.Sdk/x86_64/3.28 from flathub
[###########-        ] Downloading: 225,2 MB/416,6 MB (7,3 MB/s)   

Rate 7Mb/s

Download from ostree:

$ ostree --repo=repo pull --mirror flathub:runtime/org.gnome.Sdk/x86_64/3.26

GPG: Verification enabled, found 1 signature:

  Signature made sáb 09 jun 2018 05:14:07 CEST using RSA key ID 562702E9E3ED7EE8
  Good signature from "Flathub Repo Signing Key <flathub@flathub.org>"
  Primary key ID 4184DD4D907A7CAE

Receiving delta parts: 13/184 225,0 MB/415,4 MB 6,4 MB/s 29 seconds remaining

rate 6Mb/s

Download from buildstream: Here using https://gitlab.com/albfan/buildstream-flatpak-gtksourceview

$ git clone https://gitlab.com/albfan/buildstream-flatpak-gtksourceview
cd buildstream-flatpak-gtksourceview
$ bst build --track-all gtksourceview.bst
...
[--:--:--][????????][track:base/sdk.bst                  ] START   Fetching tracking ref 'runtime/org.gnome.Sdk/x86_64/3.28' from origin: https://dl.flathub.org/repo/
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 36% (12/33) 165.1 kB
[00:00:02][????????][track:gtksourceview.bst             ] SUCCESS Tracking master from https://gitlab.gnome.org/GNOME/gtksourceview
[00:00:02][????????][track:gtksourceview.bst             ] SUCCESS flatpak-autotools/gtksourceview/????????-track.21695.log
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 16% (77/461) 815.6 kB
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 12% (158/1250) 954.1 kB
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 8% (263/2928) 1.2 MB
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 8% (340/3992) 1.9 MB
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 10% (436/3992) 4.0 MB
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 12% (512/3992) 4.9 MB
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 15% (601/3992) 5.9 MB
[--:--:--][????????][track:base/sdk.bst                  ] STATUS  Receiving objects: 17% (705/3992) 6.5 MB
...

STATUS is updated every second so rate is as 1Mb/s

Seems that flatpak does some tricks upon ostree, and ostree python bindinds slow a lot the download

What is the current bug behavior?

A download speed 7 times slower than expected

What is the expected correct behavior?

Same download speed for BuildStream

Relevant logs and/or screenshots

See in steps to reproduce

Possible fixes

https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_ostree.py#L502

Other relevant information

• BuildStream version affected: /milestone %BuildStream_v1.1

---