Source caching is very inefficient

Summary

Trying to build freedesktop-sdk using bst master, I noticed it does a lot of IO. Trying to

Steps to reproduce

  1. In buildstream.conf, set sourcedir to somewhere outside of $XDG_CACHE_HOME (e.g. explicitly to ~/.cache/buildstream/sources)
  2. checkout branch valentind/bst2 from freedesktop-sdk
  3. run XDG_CACHE_HOME=$PWD/cache bst source fetch --deps none bootstrap/build/gcc-stage1.bst
  4. if it had to download something, rm -r cache and repeat step 3.

What is the current bug behavior?

This takes a very long time (5 min on my machine)

What is the expected correct behavior?

It should take a lot less time. Unpacking all the tarballs and applying all the patches (plus a find . -type f | xargs sha256sum after each step to simulate caching takes about 35s)

Relevant logs and/or screenshots

log using wallclock

[14:19:43][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] START   freedesktop-sdk/bootstrap-build-gcc-stage1/7b007763-fetch.503413.log
[14:21:05][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] START   Applying local patch: patches/gcc/gcc-multiarch-os-dir.patch
[14:21:05][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] SUCCESS Applying local patch: patches/gcc/gcc-multiarch-os-dir.patch
[14:22:08][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] START   Applying local patch: patches/gcc/gcc-multiarch-no-cross-structure.patch
[14:22:08][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] SUCCESS Applying local patch: patches/gcc/gcc-multiarch-no-cross-structure.patch
[14:22:54][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] START   Applying local patch: patches/gcc/gcc-do-not-override-for-build-variables.patch
[14:22:54][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] SUCCESS Applying local patch: patches/gcc/gcc-do-not-override-for-build-variables.patch
[14:23:50][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] START   Applying local patch: patches/gcc/gcc-do-not-override-for-build-variables-2.patch
[14:23:50][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] SUCCESS Applying local patch: patches/gcc/gcc-do-not-override-for-build-variables-2.patch
[14:25:03][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] START   Applying local patch: patches/gcc/gcc-disable-filesystem-ts-fix-cxx-17.patch
[14:25:03][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] SUCCESS Applying local patch: patches/gcc/gcc-disable-filesystem-ts-fix-cxx-17.patch
[14:25:18][7b007763][   fetch:bootstrap/build/gcc-stage1.bst] SUCCESS freedesktop-sdk/bootstrap-build-gcc-stage1/7b007763-fetch.503413.log

Possible fixes

This seems to be related to sources that require previous sources, as unpacking 5 tarballs (one of which is 68M) took about 1m20 while applying 5 patches took much more.

Other relevant information

  • BuildStream version affected: current master (99063c73)