Introduce Cython for better performances
Description
Introduce Cython in order to speed up hot paths in BuildStream.
This rewrites part of _yaml.py and _variables.py to be in cython, offering a good speedup compared to the
pure python version.
More rational is given on the ML: https://mail.gnome.org/archives/buildstream-list/2019-May/msg00023.html
TODO before merging
-
Ensure Source distributions ( ./setup.py sdist) do contain the cython-generated ".c" files and do not need Cython nor install it. -
Rewrite commit messages with more insight and information -
Modify Nodefile_number,col and line to be int and ensure it is typed accordingly -
Document added files and classes -
Tackle all FIXMEs -
Tackle all TODOs -
Ensure coverage for cython files
Changes proposed in this merge request:
- Introduce
pyproject.tomland isolated package builds in tox - Rewrite
_variables.pyin Cython - Rewrite
_yaml.pyin Cython - Use C when calling
_yamlstuff in_variables.py
Benchmarks
These benchmarks have been run on a lenovo T470 laptop, with 2 physical/4 virtual cores machine, with 32Go of DDR4 RAM, and a NVME SSD. The machine was running Fedora 30, fully updated as of the 22th of May, and had no other significant program running.
| commit | action | python_version | max_memory | time |
|---|---|---|---|---|
| bschubert/cython - 238458d2 | build - 4 | py37 | 176 | 150.98 |
| build - 8 | py37 | 176 | 164.62 | |
| show | py37 | 176 | 5.22 | |
| show - cached | py37 | 176 | 5.35 | |
| master - 25172ed2 | build - 4 | py37 | 185 | 156.38 |
| build - 8 | py37 | 185 | 171.02 | |
| show | py37 | 162 | 8.7 | |
| show - cached | py37 | 185 | 8.18 |
Edited by Benjamin Schubert