Skip to content

Rework the scheduler to use threads instead of processes

Benjamin Schubert requested to merge bschubert/no-multiprocessing into master

Description

This removes the usage of processes in the scheduler to use threads instead.

This is the first step towards #1389.

This implements the changes required to run with a threaded scheduler.

Things to test

I have tried extensively to stop/start/kill the processes to ensure that everything still behaves accordingly.

However, more eyes would be good. Please try breaking it!

Profiles

Two profiles to show the difference between master and now:

For source fetch:

For build

Benchmarks

Those benchmarks use the new forkserver setup for running the blocking jobs.

  • Finished in: 95 mins, 15 secs
  • Number of builders: [4, 8, 12]
  • Target files: base-files/base-files.bst
  • Number of runs: 3
  • Number of warmups: 1
  • Python versions: py38
action python_version commit median time (secs) mean time (secs) ± std
build - 12 py38 bschubert/no-multiprocessing - 2e592a38 72.18 72.61 ± 1.22
py38 master - 7fee0e64 98.12 97.82 ± 0.66
build - 4 py38 bschubert/no-multiprocessing - 2e592a38 74.38 75.15 ± 1.83
py38 master - 7fee0e64 110.59 108.34 ± 5.36
build - 8 py38 bschubert/no-multiprocessing - 2e592a38 72.93 73.54 ± 1.67
py38 master - 7fee0e64 104.23 105.04 ± 1.93
show py38 bschubert/no-multiprocessing - 2e592a38 5.24 5.27 ± 0.05
py38 master - 7fee0e64 5.14 5.35 ± 0.38
show - cached py38 bschubert/no-multiprocessing - 2e592a38 11.42 11.48 ± 0.20
py38 master - 7fee0e64 11.25 11.28 ± 0.07
show - sources fetched py38 bschubert/no-multiprocessing - 2e592a38 8.19 8.10 ± 0.41
py38 master - 7fee0e64 7.49 7.60 ± 0.32
source fetch py38 bschubert/no-multiprocessing - 2e592a38 600.05 599.23 ± 1.61
py38 master - 7fee0e64 87.39 86.86 ± 1.80
Edited by Benjamin Schubert

Merge request reports