BuildStream gets slower with a high number of runners

Summary

I've been running some benchmarks on BuildStream and have a surprising behavior.

The machine used to run the benchmarks has a 6 physical / 12 virtual CPU cores, 32Go of DDR4 RAM and a sata ssd.

the command tested is bst --builders ${BUILDERS} build essential-stack.bst from http://gitlab.com/jennis/debian-stretch-bst. Each run has been done 6 times, with the first one discarded, to take into account kernel caching. The machine had only this and minimal software stack running during the tests (systemd/NetworkManager/tmux) and was a Fedora 29 fully updated as of the 1st of march. The buildstream cache was busted between each run and the yaml cache was pre-populated.

Here is the results:

builders	min	max	mean
4	185.00	186.22	185.586
8	158.70	161.94	160.346
12	245.02	248.35	246.360

Steps to reproduce

I'm putting a way of reproducing this locally, but it's mainly running bst build, with the different number of builders.

What is the current bug behavior?

BuildStream gets way slower with 12 builders than with 8

What is the expected correct behavior?

BuildStream should be able to scale over 8 workers