Buildstream's scheduler spends a long time creating subprocesses

Summary

Buildstream seems to spend a lot of time creating subprocesses. In the below profile, of a 600s build using @danielsilverstone-ct's profiling suite, about 120s was spent in posix.fork. 3-empty-cache-build.cprofile

Steps to reproduce

Follow the instructions at https://mail.gnome.org/archives/buildstream-list/2019-January/msg00034.html as far as creating 3-empty-cache-build.cprofile.

snakeviz is a useful tool for visualising the profile.

What is the current bug behavior?

A large amount of time is spent on creating subprocesses.

What is the expected correct behavior?

Significantly less time is taken.

Possible fixes

The first step would be to have a look at where posix.fork is called and what posix.fork does, to see if there are any simple ways to reduce this time.

Failing that, look into refactoring the scheduler into a pool-based model, where there is a pool of subprocesses that are given jobs to process, instead of one subprocess per job.

Other relevant information

• BuildStream version affected: /milestone %BuildStream_v1.x

---