Introduce Cython for better performances
Description
Introduce Cython in order to speed up hot paths in BuildStream.
This rewrites part of _yaml.py
and _variables.py
to be in cython, offering a good speedup compared to the
pure python version.
More rational is given on the ML: https://mail.gnome.org/archives/buildstream-list/2019-May/msg00023.html
TODO before merging
-
Ensure Source distributions ( ./setup.py sdist
) do contain the cython-generated ".c" files and do not need Cython nor install it. -
Rewrite commit messages with more insight and information -
Modify Node
file_number,col and line to be int and ensure it is typed accordingly -
Document added files and classes -
Tackle all FIXMEs -
Tackle all TODOs -
Ensure coverage for cython files
Changes proposed in this merge request:
- Introduce
pyproject.toml
and isolated package builds in tox - Rewrite
_variables.py
in Cython - Rewrite
_yaml.py
in Cython - Use C when calling
_yaml
stuff in_variables.py
Benchmarks
These benchmarks have been run on a lenovo T470 laptop, with 2 physical/4 virtual cores machine, with 32Go of DDR4 RAM, and a NVME SSD. The machine was running Fedora 30, fully updated as of the 22th of May, and had no other significant program running.
commit | action | python_version | max_memory | time |
---|---|---|---|---|
bschubert/cython - 238458d2 | build - 4 | py37 | 176 | 150.98 |
build - 8 | py37 | 176 | 164.62 | |
show | py37 | 176 | 5.22 | |
show - cached | py37 | 176 | 5.35 | |
master - 25172ed2 | build - 4 | py37 | 185 | 156.38 |
build - 8 | py37 | 185 | 171.02 | |
show | py37 | 162 | 8.7 | |
show - cached | py37 | 185 | 8.18 |
Edited by Benjamin Schubert