Buildstream's scheduler spends a long time creating subprocesses
Summary
Buildstream seems to spend a lot of time creating subprocesses. In the below profile, of a 600s build using @danielsilverstone-ct's profiling suite, about 120s was spent in posix.fork. 3-empty-cache-build.cprofile
Steps to reproduce
Follow the instructions at https://mail.gnome.org/archives/buildstream-list/2019-January/msg00034.html as far as creating 3-empty-cache-build.cprofile
.
snakeviz
is a useful tool for visualising the profile.
What is the current bug behavior?
A large amount of time is spent on creating subprocesses.
What is the expected correct behavior?
Significantly less time is taken.
Possible fixes
The first step would be to have a look at where posix.fork is called and what posix.fork does, to see if there are any simple ways to reduce this time.
Failing that, look into refactoring the scheduler into a pool-based model, where there is a pool of subprocesses that are given jobs to process, instead of one subprocess per job.
Other relevant information
- BuildStream version affected: /milestone %BuildStream_v1.x