Skip to content

Introduce Cython for better performances

Benjamin Schubert requested to merge bschubert/cython into master

Description

Introduce Cython in order to speed up hot paths in BuildStream.

This rewrites part of _yaml.py and _variables.py to be in cython, offering a good speedup compared to the pure python version.

More rational is given on the ML: https://mail.gnome.org/archives/buildstream-list/2019-May/msg00023.html

TODO before merging

  • Ensure Source distributions (./setup.py sdist) do contain the cython-generated ".c" files and do not need Cython nor install it.
  • Rewrite commit messages with more insight and information
  • Modify Node file_number,col and line to be int and ensure it is typed accordingly
  • Document added files and classes
  • Tackle all FIXMEs
  • Tackle all TODOs
  • Ensure coverage for cython files

Changes proposed in this merge request:

  • Introduce pyproject.toml and isolated package builds in tox
  • Rewrite _variables.py in Cython
  • Rewrite _yaml.py in Cython
  • Use C when calling _yaml stuff in _variables.py

Benchmarks

These benchmarks have been run on a lenovo T470 laptop, with 2 physical/4 virtual cores machine, with 32Go of DDR4 RAM, and a NVME SSD. The machine was running Fedora 30, fully updated as of the 22th of May, and had no other significant program running.


commit action python_version max_memory time
bschubert/cython - 238458d2 build - 4 py37 176 150.98
build - 8 py37 176 164.62
show py37 176 5.22
show - cached py37 176 5.35
master - 25172ed2 build - 4 py37 185 156.38
build - 8 py37 185 171.02
show py37 162 8.7
show - cached py37 185 8.18


Edited by Benjamin Schubert

Merge request reports