Refactor _update_state() to be called only when needed
Description
This changes when we call _update_state() to be called everytime it might be needed but not otherwise.
Please review very carefully as this is touching very important parts of the codebase.
I will add performance benchmarks. Currently, up to 40% improvements has been noticed.
Proposed changes in this MR:
Changes proposed in this merge request:
- Store all reverse dependencies information
- Don't call update_state() on each status() in Queues
- update reverse dependencies states on pull and workspace builds
When merged, this MR will close #703 (closed)
Benchmarks and results
On a simple project (45 bst files, 2 dependencies per element, 1 empty file per element) when running "bst build without remote caches":
Master time: 2.173143, Test time: 1.784614, Difference: -0.388530, Percent: -17.878700%
On a bigger project (5994 bst files, 2 dependencies per element, 1 empty file per element) when running "bst build without remote caches":
Master time: 1326.898480, Test time: 807.525059, Difference: -519.373421, Percent: -39.141911%
Those previous benchmarks were done on a broken version of master. The current state is not as good.
Results can be reproduced by running ./benchmark.py bschubert/pipeline, found in https://gitlab.com/BenjaminSchubert/buildstream-benchmarks
New sets of benchmarks
These benchmarks have been run on a 6 physical/12 virtual cores machine, with 32Go of DDR4 RAM, and a SATA6 SSD. The machine was running the latest version of fedora 29 as of the 1st of March, and was kept with the minimal running (systemd/NetworkManager/the benchmark). Each run was done 6 times, with the first run of each discarded on essential-stack from http://gitlab.com/jennis/debian-stretch-bst
The results are in the following table. We can see that starting from 8 builders, we can see a difference in performances.
Master was at version 96426103 at the time of benchmarking
| step | branch | interpreter | clock_time | max_memory (MB) |
|---|---|---|---|---|
| bst build - 12 builders | bschubert/pipeline | py37 | 227.66 | 467 |
| master | py37 | 243.56 | 464 | |
| bst build - 4 builders | bschubert/pipeline | py37 | 200.33 | 467 |
| master | py37 | 197.48 | 465 | |
| bst build - 8 builders | bschubert/pipeline | py37 | 192.85 | 467 |
| master | py37 | 199.59 | 465 | |
| bst show everything cached | bschubert/pipeline | py37 | 19.5 | 595 |
| master | py37 | 19.58 | 592 | |
| bst show with cache | bschubert/pipeline | py37 | 14.51 | 466 |
| master | py37 | 14.39 | 464 | |
| bst show without cache | bschubert/pipeline | py37 | 27.54 | 445 |
| master | py37 | 27.6 | 443 |