Rework the scheduler to use threads instead of processes
Description
This removes the usage of processes in the scheduler to use threads instead.
This is the first step towards #1389.
This implements the changes required to run with a threaded scheduler.
Things to test
I have tried extensively to stop/start/kill the processes to ensure that everything still behaves accordingly.
However, more eyes would be good. Please try breaking it!
Profiles
Two profiles to show the difference between master and now:
For source fetch
:
For build
Benchmarks
Those benchmarks use the new forkserver
setup for running the blocking jobs.
- Finished in: 95 mins, 15 secs
- Number of builders: [4, 8, 12]
- Target files: base-files/base-files.bst
- Number of runs: 3
- Number of warmups: 1
- Python versions: py38
action | python_version | commit | median time (secs) | mean time (secs) ± std |
---|---|---|---|---|
build - 12 | py38 | bschubert/no-multiprocessing - 2e592a38 | 72.18 | 72.61 ± 1.22 |
py38 | master - 7fee0e64 | 98.12 | 97.82 ± 0.66 | |
build - 4 | py38 | bschubert/no-multiprocessing - 2e592a38 | 74.38 | 75.15 ± 1.83 |
py38 | master - 7fee0e64 | 110.59 | 108.34 ± 5.36 | |
build - 8 | py38 | bschubert/no-multiprocessing - 2e592a38 | 72.93 | 73.54 ± 1.67 |
py38 | master - 7fee0e64 | 104.23 | 105.04 ± 1.93 | |
show | py38 | bschubert/no-multiprocessing - 2e592a38 | 5.24 | 5.27 ± 0.05 |
py38 | master - 7fee0e64 | 5.14 | 5.35 ± 0.38 | |
show - cached | py38 | bschubert/no-multiprocessing - 2e592a38 | 11.42 | 11.48 ± 0.20 |
py38 | master - 7fee0e64 | 11.25 | 11.28 ± 0.07 | |
show - sources fetched | py38 | bschubert/no-multiprocessing - 2e592a38 | 8.19 | 8.10 ± 0.41 |
py38 | master - 7fee0e64 | 7.49 | 7.60 ± 0.32 | |
source fetch | py38 | bschubert/no-multiprocessing - 2e592a38 | 600.05 | 599.23 ± 1.61 |
py38 | master - 7fee0e64 | 87.39 | 86.86 ± 1.80 |
Edited by Benjamin Schubert